Really good question. And he left her to suffer from the temple, from her family... What a good catch... oh, wait...
You get it when reading the novel but you loose it in a manga because they are trying to show something that would likely take 2 chapters or more to cover in pictures but is a few paragraphs long in the novel. The prince has issues with having his life being dictated to him. He never has a say so in his choice in marriage and he was taking it out on her. The has a massive superiority issue going on. He has complex issues regarding how he sees the female lead. Later in the novel you find out he is upset because she can't move past the issue with the child/slave boy dying in her mind thinking she is thinking too much of this other person. We learn he is in fact wanting her to come crawling and begging back to him for her help and to have her marry him. The prince as I said has a twisted mind of being superior that he eventually wakes up from at the end of the novel.
If he knew all along what's going on, what was that in chapters 7-8?