I dunno I think that one flashback to after their high school graduation wasn't as pronounced to properly show this. But my understanding (could be wrong):
- he noticed Igarashi liked him and thought that cute (but nothing more, which isn't exactly wholesome)
- then the guy who made him realize he always had a thing for tall, black hair short haired girls
- and then the hunt for Igarashi
I did NOT understand his thinking of "didn't mean to reject you" but I can understand his reasoning of "he cut off contact after graduation on his own terms and without going back on it, if he bails NOW it'll probably be to the end of the world including a nose-job and a name change"

i dont get shida
so he knew igarashi liked him
he also liked igarashi back
but he was scared?? scared of what exactly
he ended up dating girls and even had sex with igarashi but didnt confess
lame