He either is bi or gay and won't recognize it or maybe it's like so many yaoi mangas where the mc only has a thing for the other mc guy.
I thought the anger was about the lust. Mainly because it could, if he gave it enough time and consideration, turn into love and that he doesn't want to get involved with this guy. Mostly because this is just supposed to a temporary job to handle the debts and also because the other guy is a prostitute and that is just asking for a world of hurt and broken hearts.

Is Tora really straight or did he lie/fudge the truth so they would leave him alone?
All he says is 'I'm not gay.' That doesn't mean he can't be bi or pan.
And he certainly has a bit of a lust for the guy.