Bruhh

You have to remember that this is a different era. While yes, the FL/ML were on the younger side of things, it wasn’t unheard of for nobles to take over at a young age. And while it wasn’t described in the story, I’m sure the Duke at least checked in periodically to make sure that they were doing alright (unless you want the story to be like Lady Baby and be stuck in their childhood forever and a day lol).

They asked the kids if they wanted to go and they said no... Did you want them to force the children to go? Staying wasn't an option at that point because they needed to work on themselves as a couple. The children proved to be capable which is probably why they didn't go. I'm not really excusing their behavior because that was shifty but I digress

I think there is some difference in the interpretation due to the machine translation of the work because to me it seemed like the Parents were the one to went back to their estate or dukedom while the kids remained in the capital to learn their duties more and run their side project businesses. The main lead won't be able to meet the Prince that frequently if they weren't in the capital. And as another reply pointed out, the parents actually asked the kids if they wanted to come with them or not. I won't they say they are exceptional parents but they didn't seem red flag like either, at least to me.