I have theory that he did love her in her first life, it was just she was such a bad person that he didn’t wanna accept and acknowledge his feelings cause of how she was back then, one of the tell tale signs that he did love her is that he never really apprehended her on her actions, he just let her do what she wants never reprimanding her or telling her off, and I also think that he could and can make her authority as an empress useless or be lessened if he wanted to but he just couldn’t, I think he felt some sort of guilt or its just that she has such high standing even he couldn’t do anything. Anyways this is just my theory, you can agree or disagree.
Pls don’t forget that she was a very very bad person in her first life, so it’s pretty understandable if He didn’t love her then or atleast showed it. She was straight up an evil person, she was mistreating and abusing people left and right, using her authority to get away from it, so him not being affectionate isn’t that surprising honestly. The fact that everyone just ignored it and never tried correcting her behavior, people around her is also at fault for her becoming like that. She literally abused her brother his whole entire life and not once her father stopped or talked to her about what she was doing.