Wait...

Well im guessing its the typical cliche ...... let's see if this was the original FL then I'm guessing after he told her not to expect love from him she agreed cuz she loved him but that didnt stop her from trying to get his affection so for him that was just annoying him..... and then the "heroine" came and she wasn't a busy body or annoying girl so he fell in love made her the empress and our FL got banished.... you know those tropes....but our girl really doesn't love him so she stayed away like her behavior screamed I really wont bother you cuz I'm not intrested........ so from this i gather his the type that if you push he will back away but if you pull away he will come to you....... so yeah since she pulled away he found her interesting and I'm guessing since she has no motives to actually love him she was always sincere and treated him normal so he probs liked that....... and you know there's obviously a backstory for him that explains why his so emotionally constipated....(⌒▽⌒)

Gonna sound cringe. But love can't be explained in real life, so the authors make the plot based on that. This is much more realistic tbh. And i lobe it.