The dad is dumb

The dad is dumb, but he couldn't direct his anger at the royal family and other ducal family, so he directed it towards the easiest target, Olivia.

I also have to mention, he is actually justified in his anger towards Olivia's mother. Unlike other manhwas, the father got legitimately raped by Olivia's mom. Olivia's mom was coerced and blackmailed to sleep with the Duke by the duke's political enemy. Since he is part of the imperial(?) faction with impeccable reputation, known to he a devoted and loving husband, they targeted his reputation as a faithful husband.

They drugged and knocked him out during a party, and had Olivia's mom sleep with him.