Both, both is good.
You should hate the duke more. Hope he die first in a brutal way.
The Duke, totally. He's the one who went out his way to force-take his illegitimate kid just to basically condemn him to years of abuse while a the same time taunting his legal wife of the child's existence. I'm sure the duchess won't do anything if the duke didn't bring the child to her home.
Don’t know who I hate more. The duchess who abuses him or the duke who allows it to happen.