The one voicing those thoughts is the emperor, and the duke is someone who used to be another member of the royal family and elligible for the throne, but lost the competition with the current emperor, That duke is the father of the head maid that serves the blond princess (the maid that gets called "Princess Ivanya" even though she's not an "imperial" princess), and it seems both father and daughter are related to the royal family and may have been in the line of succesion but where taken out in favor of the current emperor, for some reason that we don't know yet. Keep in mind there are kingdoms and times where the firstborn prince isn't necessarily the one who actually gets to sit on the throne, like for example in kingdoms where the custom is for the king makes all his sons compete with one another in order to choose the most efficient one as the next ruler...

The translation is so weird. Who's duke? And what abt the princess? Is that her father or her uncle? Who lost the throne to whom?