You got it … only after telling the man “I think it’s best to call it an evening now” ask if he’s in agreement. If he is, great. If not, then say something like, “I’m sorry you don’t agree. I don’t feel comfortable staying any longer and I must leave.” This gives him the opportunity to express his thoughts and not be so one sided. Hope this helps.
Love, Cherry