I think the worst thing was not communicating with FL better, back on earth his hands and feet was practically tied up, he was a hostage on earth, how do you expect him to protect FL when he barely could protect himself from those who tried to take advantage of him, he was probably tied to some unbreakable unfair contract or oath.
that was why he was so paranoid all the time and can't show any sort of feelings for FL even if he does, because FL will become another hostage against him.
Makes sense. This is a unique scenario where the dude is actually not in a novelesque place isekaied but actually just time travelled back to his place.
It is a shitty situation before. Only thing left is to redeem oneself.
Oh so you knew FL was being bullied by that saintess b*tch. I bet you just stood there and did nothing about it. I hope FL makes you suffer much much more before letting you into her life again.