I think maybe it's because his EQ/social interactions is a bit stunted, and he has low self-esteem. He doesn't really think that someone that popular and cool could like him back or want to be with him in the long term, and that the ML is just doing this out of obligation because he's kind.
I think it's due to his past experience. The way Hyunwoo talks about his mom makes me think they aren't on talking terms anymore. Him not being able to discuss ghostly stuff around her eventually closed off their relationship. I assume he believes that his and Jiho's relationship will eventually end in a similar fashion since there isn't anything necessarily tying them together
Why does he think they'll drift apart!?