Thankyou for the explanation but I still have some questions like is the ghost is some kind of succubus..... that girl, why can she see the ghost....... isn't the manager and the brown hair already in relationship or is it just one sided love from brown hair........ does MC still doesn't know that he was sleeping with a ghost.
From what i have roughly understood:
1. the girl has the ability to see ghosts (like a sensitive or something, she said so while she was in the elevator with the brown haired boy, that was making her nauseous)
2. The manager and the brown haired boy have a reciprocal relationship (or something like that) so it's not one sided, but recently the brown dude was being too attached so the manager was getting a bit annoyed
3. The mc still doesn't know that the person he's sleeping with is not the director but the ghost
I know this explanation is a bit messy so if you have any other questions feel free to ask!
In the beginning, the ghost was after the brown haired boy since he was masturbating on his work desk, but after getting caught by the mc, the ghost rapes him even though he was not his type, to discover he was really good in a sexual way(?), like after forcing the mc to blow him, he enjoyed the job and got interested.
So as you can see now the mc is being hunted by the ghost since then.
To clarify, i think the ghost was doing the same thing with the brown haired boy initially, in fact he was together both with the real manager and the fake one, so he got exhausted (because has you have seen the ghost can posses the characters even during their sleep or something like that, so basically they never rest) and fainted at work. Afterwards he gets back since he finally rested, and that's probably because the ghost was hunting the mc in the meanwhile. Also i believe there are two ghosts in the building, and one of those might be possessing the brown dude sometimes.
Don't trust me much about the last two parts coz im not so sure but that's what i understood, i could be wrong tho
I don't think it was so much that MC is "great at sex" but that he's fearful (note ch 1: he's scared shitless when the women are gossiping about someone seeing a ghost).
The ghost seems to get fuel from strong emotions. Yandere like love from brown hair was a worthwhile substance, but it could not compare to overwhelming fear that MC experiences.
MC doesn't love the manager. He fears him. So much so that he stutters and beats around the bush when trying to confront the very first r*pe that happened. It makes sense that even as the ghost reveals it's true form, MC can't see past his own fears.

For those who are still confused (spoilers)
The ghost is taking the form of the manager, it isn’t actually the real manager that’s doing this stuff to mc.
That’s why in one of the chapters when mc tries to confront the real manager about it, the real manager doesn’t know wtf mc is talking about. On the latest chapter 34 is when the real manager comes back, it’s not the ghost. And real manager and the brown haired guy that always clings to the real manager had a “relationship” or something, and the ghost took advantage of that in the beginning so the ghost would take the managers form to do naughty stuff with the brown haired guy, that’s until mc walks in on the ghost that has the managers form, so now the ghost goes after mc. While the real manager has nothing to do with it. Until On the recent chapters the real manager had a dream about our mc, so now the real manager is a bit interested in mc. The ghost can take different forms, but he mostly takes the form to look like the manager. Also you can tell the difference sometimes because the real manager has softer tone in skin color and has a more soft look in his eyes. But when it’s the ghost taking the managers form, then he will have more pale skin and yellow eyes. (Hope this helps)