As in your case, it happened because we forgot to latch the door. Fortunately, the hotel staff didn't come all the way into the room so they couldn't see me. They saw Mistress, however. She told them the room was occupied and they quickly departed.
So it was more than one person? Did they knock? Say hello? What time was it?
I'm wondering if some of these no-knock, no-hello "drop ins" are actually staff spying on people they suspect of things.