Your theory sounds, well… sound. I’d like to propose a counter theory.
I don’t think hajin was rejected for bullying his younger bro. Because she was already rejecting him during his drowning incident. He was a sweet, very young child then- yet she wanted him to die… to me that isn’t momma bear behavior. Letting a young child drown, who loves you so much, and is so sweet… what could he have even done to his brother, at that age, to warrant death?
Are we sure he’s her biological son? He looks like the dad, but maybe that’s why the dad didn’t question it? If the pregnancy with hajin’s dad was real… but she gave him a fake baby (hajin) so she could flee overseas with her real baby… then that would make sense. I think she just wanted to escape from him.
The affair rumors are because people think she’s dating her son. They think she’s a cougar. Hajin’s bro is the real son. Isn’t that why she said, “you know what that child is to US” rather than “to her”. The dad might not want to reveal this mess to the public. That’s why he’s chosen to reject his biological son. That’s also why the bro told hajin, “ you stole everything from me”
That line about “you know what the child is to us” was about the illegitimate child and is essentially her collar to be a good wife. The condition of her being a good wife is probably that father ignores the mama-son dates and doesn’t hurt him. The father is so concerned with appearances, he would NEVER let an illegitimate or swapped son into his home. It just wouldn’t happen. We know for certain that Hajin is his son. I don’t believe Hajin would have been too young to hurt his brother. My older cousins used to absolutely terrorise me and they were 5-7 while I was 2-3. You can be a douchebag at any age and still be sweet and loving to the adults. Also Hajin does look like the father and although yes he doesn’t look the same as mama, if he was an illegitimate child, Hajins father would have hajins real mother nearby somewhere and Hajin would have known about it. She would have been his nanny and nursemaid. The affair probably did happen though. I know THIS affair report is about the son but I don’t believe that they were raised together and the only reason for that would be that he’s an affair baby.
Also about the drowning, a mother to a newborn can kill her baby even if she loves them very very much if she’s sleep deprived and in a survival state of being. A father can too but mothers all over the globe have snapped and killed their own precious babies due to this phenomenon of being so stressed and tired, they just couldn’t handle it. If she didn’t like hajin from the beginning because Hajin is the son of the abusive husband she has, it would make sense that in a high stress situation (her marriage), one small trigger can have her kill her son or just let him die. It’s not uncommon actually.

Hajin was born first. The father told the mother that she had to get pregnant and do her job as a wife or she’s done for (presumably she’s been having an affair and he threatened the partner). Mother found out she was pregnant with the affair partners child when Hajin was around 4-5 and had the baby. Instead of letting people know that she cheated and had a bastard child, daddy dearest forced her to stay in England with the kid for a bit and then dumped the child onto the grandparents and mother came home without the newborn after father threatened to do something with that child and her partner. Her affair partner is most likely dead. Hajin being the legitimate son and the eldest probably took after papa in his cruel ways and bullied the illegitimate younger half-brother. Mama bear did not like that and started seeing hajin as daddy 2.0 and was NOT happy. So she hates hajin for what he represents. Her golden baby boy could do no wrong because that’s her lovers son. The whole “swap” thing is about their stories in England. The butler said that Hajin was the one who lived with the grandparents in England but it was actually the illegitimate brother that was in England.
I think the mother did something wrong in the past before Hajin was born and that the father is holding it over her head like a guillotine ready to come down at any moment that’s he decides he’s had enough of her insubordination (because he treats her like his lessor rather than his equal). He probably abused her mentally this whole time with the fact that he could easily kill the last remaining trace of her dearly departed lover whenever he wanted to and that it’s by his graciousness that the child lives.
If you think something is wrong with my theory or timeline, feel free to correct me but I’m pretty sure this is it.