I don't think he knew they were being abused by that woman before he saw them(if i remember right)? And he sent them back to that place cuz the evil duke convinced him that the children were uncomfortable in the palace, and would be happier if they went back to the place they spent their childhood in, away from the royals...the emperor isn't completely blameless but i do think he tried to be better to the kids after bringing them to the castle
The red guy told him how they were being abused
that was after the mother died i think
So he left them with her for years while knowing they were abused cause the red hair guy told him?? And he thought it would be good to let them stay at the place they were abused and neglected without asking them if they wanted to???