I guess it means that he's sexually frustrated cause he wants to be nice to Yuuji so he tries to reserve himself. The spirit appears with sexual frustration, you can still be frustrated even if you were getting some. This was my understanding anyway. ヾ(❀╹◡╹)ノ~
Why is the sex spirit/ghost thing still there?