I think that this is very related to the opposite question, when you decide to move in. If you see a mistress once does it count? I would say that the general answer is that when the reason you decided to move in does not hold any longer.

My two cents for you, if you like to see many mistresses why move on at all? What does it mean anyway, since you are seeing any particular mistress only occasionally? Your mood or practical circumstances might change, so you could just see any particular mistress more or less often. You don't need to make definitive decisions.