Responses
Spoiler ahead:
Damian is not his son, when his brother died the Duke was just a teen suddenly found out about his naphew. And since yo continue the blood line is such a disgusting process, he already had his heir.
And the Duke receives daily reports of Damian, every single detail.
And yes he could be more affectionate, but i he hates his bloodline so much that he acts like that. I know, again, doesn't justify the fact that he could be a little more kind to the kid since it's not even his fault. But i think i now understand his character more.

I'm not for giving the Duke for neglecting Damian and not giving him any attention.