As in most cases of communication, honesty is the best policy. A simple explanation that you are declining future opportunities is best delivered with a healthy does of honesty along with a declaration that no one is "at fault" for this decision. Telling a sub that you do not think that you are suitable play partners can be cushioned by telling the sub that there are plenty of other Dommes who would be willing, if not eager, to play with him. I think the key here is not to assign blame for the incompatibility but to just say that you want him to be happy with his next session that is more likely to happen if he sessions with another Domme. Goodbye and good luck in a gentle and positive send off.

I would rather a Domme tell me after the first session that we were a bad combo than invest hundreds of dollars in the next session and to then to find that out.

There is never any excuse for stalking a Dommme after a session, for any reason at all. There have been times for me that I had an initial session with a Domme after participating in all good faith and then realized we were incompatible for any further play. I didn't blame the Domme, or even myself. I moved on, and I would hope that the vast majority of subs would respectfully do the same if the roles were reversed.