If the prince had to just marry one of them, why couldn't he just stick with the first dau...

with what was show of him i can only think of one reason, his inner desires were showing, his preferences are for younger ones -_- he was hoping for a loli love... or something like that LOL
but yeah, it is really confusing if he is so worried about appearances and he choses to change fiance out of no where, without even checking the girl before. for someone who look sensate it is such weird move, if he loved her at least that would be a excuse, but no... he was dumb by free choice.

I think in the earlier chapters it was mentioned that the marriage between him and the younger sister was offered, and an annulment between the older sister and the prince. Because the bitchy little sister wanted her older sister's betrothed. Because the prince thought they'd be similar, and because he needed to wed someone from the duke's family, he accepted. Which promptly resulted in this situation.

The prince could care less who would be his fiancee as long as it was from the house. His standing as a crown prince mattered more and he can't afford to lose to the second prince; who has more backing and is a threat to his status which is why he has to marry someone from a powerful family. There was nothing stated regarding his desires. Anyhow, their relationship is the least of my worries. ╮(￣▽￣)╭

Also I feel like it is known that the mc's family likes the 2nd daughter more than the first daughter. So I think because of this he agreed to the change of fiancee. Also as stated before, he didnt know what she was like but he thought the second daughter was similar to the mc, and just needed to marry into the family so he didnt care.

They did have an engagement with the mc but mc's @sshole of a father wrote to the prince if he can switch fiance, since the main reason for the engagement is for the duke's power the prince thought it would be fine as long as he gain the duke's backing.

Now he regrets it, serves him right!