Leave, he's serious he doesn't want you there. There's a reason he wanted men with swords to be around when he asked you to leave the Manor.
