I think he use dark magic(?) To reverse time in the early chapter. I think that's the reason why the holy sword keep rejecting him now. (This is what I know)
Actually I kinda forgot what happened at early chapter. So I'm sorry if I confused you or maybe give you a wrong information.
Oh yes I remember the black magic. Hmmmm. But why it happens after they went to ruins
If I remember correctly. He ate the demon king heart to reverse time to safe the duchess. So now he has darkness within him. That why the sword which responds to light is rejecting him.
I'm more concerned of the Duke. What happened to him in the ruin. What of his ability to hold the sword was transferred to his wife. Or maybe his ability was sealed.