He may be feeling betrayed because, if I'm understanding this right, you never verbally told him you didn't like it and you felt/feel uncomfortable with it. I think you should explain to him that you are uncomfortable with it and don't want him saying it. (Maybe explain why he shouldn't as well and see if he will listen).
If he doesn't you may wan......
