It’s been a while for me too, but from what I remember :
She abandoned both hyesung and his father which then resulted in the dad resenting hyesung growing up especially bc of their similarities. She only appeared years later after he (hyesung) was already married and well off. She started anew and came back when it was convenient. It causes hyesung to spiral every time (rightfully so). That kind of feeling must be terrible. Hyesung has a lot of issues and insecurities that stem from this abandonment. He felt unwanted for a good portion of his life, so having people close to you in contact with the one that hurt and scarred you the most is probably the worst level of betrayal for him.

I will NEVER blame hye-sung
sure, he had some moments where he wasn't totally in the right, but he didn't WANT Byul, Dojin had to bribe him to even have give birth in the first place. Byul is a child who doesn't know the impact Hye-Sung's mom made on him. The person truly at fault here is Dojin. He's seen first hand what happened when Hye-Sung's mom tried to come back into his life, yet he still kept fucking contact with her?? Byul is a child who doesn't know much abt the topic, but Dojin is a full grown man who knew exactly what happened. Sure, I'm pissed Byul was talking to his grandma while knowing at least the basics of what happened, but Dojim keeping contact is unforgivable imo.