I just think the father can't handle too many things happening all of the same time. Not that he didn't care about Haruki but it's just that not only he lost his wife but also their baby.
^^^ This.
You do have a point, but it's not only now. He didn't do anything, when the mother threw water at Haruki. And he didn't even notice the fact that his son was at the point of breaking.
That's typical of men, in my opinion. He's the kind of father who is not expressive and is not keen on details. But I imagine if he only had to worry about one thing, Haruki, and not have to deal with Haruka and their baby's death, he would've been there with Haruki.
There are fathers out there who just doesn't care about their sons when they are already at a certain age because they expect them to already be able to handle themselves and only care when there is grave situation happening, like if he was on an car accident or was shot. He's not an admirable kind of father, don't get me wrong but I think he does care for his son just not expressive as we want him to be.
I don't like the father. He doesn't seem to care about Haruki at all. The mother was annoying, but at least she cared about her child.