I can't be sure about the guy but your experience of "I don't remember the last time I was mean to him" Or "the joke I said wasn't even that serious" Etc sounds like a heavily self biased pov.
And you being mean to others is a very valid reason for breaking off a friendship. Being mean to others means others hate you/dislike you and being associate......
