Because based on what I have read, they had a son. Karina still has 5 years to live(in our current chapter) and at 3 years old, something atk their castle(not sure if it was a monster or a human), but they targetted the child and Karina save him(their child). With that, Karina's 5 years of life ended because of severe damage. Therefore, he started to neglect their child because he blames that it was their child's fault that Karina died.
But then again, this or not. I'm just saying
Based on the spoilers I have found no such things happened(ML didn't neglect their child).....they had a daughter not a son....I did read a spoiler which said that in order to save their daughter, FL's life span decreased(or she died)......but there was no mentioning of ML neglecting their daughter because of that....besides there was also a spoiler showing how the FL had reached out to the ML after her death and she was telling how he was raising their daughter well ...

I really love this story, but I wanna drop it because of the certain spoiler I read(in which hopefully they change it).
Based of I remember from the spoiler, the ML would neglect the baby after she passed away since their child was the reason why the FL would die(I think the child was 3 years old at that time and there was an incident or atk. Not sure, but that's how the story flows).