Well, part of it is just his personal feelings. And to a degree, he's right. Even though it would reflect badly on those around him, he kept those aspects. No matter if those others accepted it or not, he knew full well the way he was reflected badly on others, especially in a country like Japan, where eccentricities are often considered a bad thing. Or who you associate with must be a total reflection of your character, even without knowing you or that person. It is a country and culture of prejudice, I can tell as much from many aspects of the culture we get to see through manga.
So personal feelings or not, he's not necessarily wrong.

I’m not really sold on the whole “blonde hair and earrings was selfishness thing.” maybe it’s the translation thing.