Because he believes he has no right. This time it was your choice whether to go back or completely cut him off of your life.