Free? Free for other people to rape him again with that feminine face of his? Free for other people to ridicule him again once they knew he was a boy toy? Free for other people to slave him knowing he was previously a noble?
He hasn't even finished his education, how would he survive out there? What jobs would he get for himself, making sure that the aforementioned would be avoided? He doesn't have any talent he can profit from either. That kinda struggle.
But it wasn't captivity though, when he was "playing the part" and being nice. He had a decent level of freedom, because the seme had developed trust in him. And it was clear even though he wanted revenge, the treatment from the seme had him conflicted cause he ended up catching feelings.
I just want everything to go back to that time, author needs to throw in some amnesia for the uke or something. I wanna see their cute fluffy times again.

Governor has a point, though. You want Yeonjo to live out there, struggling? Only because he felt a traitor and that he betrayed his family for falling in love to the very man who ended them? To punish himself when he has every right to live a normal and happy life again. We can't deny the serenity they both felt ... Aigoo.. what a cruel world ..