Very interesting topic. I think the general answer is go with your feelings. There have been two situations in my life when I met a partner who I was seeing to be head and shoulders above the rest, at least for me.

I never made exclusivity promises, but the truth of the matter, is that at least for me, time and money are both issues, so you need to prioritize. If I could spend more time on this I still think that the best way to do it would be with my mistress.

Of course this is a personal issue, and part of the answer is what is more important to you, activities or feelings.