Turn the ringer volume down or off. Caller will hear the phone ring and then go to voice mail.

If you have no other excuse you can always just blame your carrier -- "I saw your voice mail but never got the call!"