
According to some spoilers, his mother was receiving disturbing gifts at her bedroom door and thought it was Karhan. She confronted him about it but he denied ever doing such thing and she didn't believe him at all. That caused heavy tension in the house and animosity.
Even more spoiler ....
It was his brother that did such things and his mother, knowing she was deeply in the wrong for how she treated his youngest baby, decided to leave quietly after apologizing (if I recall correctly) and went to another mansion either in the countryside or South. She knows what she did left a permanent stain on the mother-son relationship, so gave her blessing to the couple and rights as heir before leaving as well.
What's the reason for them to ghost him? he's so pure and innocent