That is the Main reason people hate him. A male protagonist with a cold/distant personality and or dislike of women due to past experiences is nothing new and if anything that's what like 90% of romance reincarnation mahnwa/manga start like. The thing with this story is that noah is the apple of the story reader's eye so it only fuels the aversion towards the duke.
Yeah, I imagined that might be why he was neglecting Noah as well, so I didn’t really blame him for it. What I really didn’t like though was that he didn’t even show MC basic manners. Yeah, he can dislike women because of his trauma, but MC hadn’t even done anything. They barely interacted but he was super rude to her. Even when you don’t like someone you can still be polite to them and acknowledge their presence. She IS his legal wife after all

If the Duke’s backstory is that he was sexually abused and that’s why he behaves the way he does, then here’s my take:
I’m pro-choice and always have been especially as someone whose parents weren’t emotionally ready to be parents. If a woman is raped, forced to keep the child, and struggles to be a good mother, I don’t blame her. None of it was her choice. Is the child at fault? No, of course not. But neither is the mother who was forced into that situation. And now she's expected to care for the child of their abuser (again, child is not at fault for this)
I’m bringing this up because it feels like people are glossing over the fact that the Duke may have been abused and is now expected to parent a child he didn’t want, all while carrying unresolved trauma. Even with therapy, healing from something like that isn’t quick or easy.
The MC even said that all of Noah’s needs were met his father just didn’t interact with him, which isn’t the same as physical or verbal abuse. I don’t think Noah deserves that distance, but I also can’t bring myself to fully hate the Duke. I just hope he can heal not just for Noah or the MC, but for himself.