
Depends on how you view a harem. There doesn't seem to be any romantic feelings from the main male character for any of his female subordinates. He only seems to view them as valued comrades and friends. They seem to worship him to an extent, but not sure if it is romantic or more of a a diety sort of worship, as in he is the greatest and there is none better.
╮( ̄▽ ̄)╭ basically, I'm not getting any romance vibes out of this harem, so I don't really see it ruining the story in that way.

I mean what I'm about to say with 100% sincerity, and no sarcasm or attitude: please show me where the f**k him sentiment parts are. It's been a long time since I read the earlier parts of the story, and I don't like being uninformed, so if that type of feeling is there and I missed it, I'd like to know so I'm not giving out wrong information when I comment on someone's comment and say there isn't anything like that.
Please and thank you. I appreciate you bringing that to my attention so I can correct myself if necessary. Σ(  ̄□ ̄||)

Worship him, yes. Screw him? Nah, that's only a handful of them. They worship him as their Lord, King, etc. for being able to bring them into their being. It's a mechanism of the summoning done from the cards he gets through the Gacha Gift so that they don't betray him. If he dies, then their purpose of being would just dissipate in the end.

I just don't like them; they're like Magical Artificial Intelligence made in flesh. Sure, they each have their own unique personality and different levels of intelligence, but they all have the same programming of obsessively wanting to serve him like a God. I just feel like if someone were to have the ability to steal his Gacha gift, all of his magically created Subjects would either disappear or turn against him in seconds, even try to kill him for not bowing to the new wielder of their Gacha gift Master. Their loyalty to him seems to be under the condition that he wield the Gatcha Gift.

If his ability is stolen, I'm sure the ones he'd summoned would still obey him. Now it's an off the wall theory if it did happen, but I'm leaning more towards that if it was stolen the cards that one summons would be what obey that one, but that's if they didn't realize the one key component to make the cards summoned through the Gacha actually pop out with anything but trash rank. It's the reason the base is at the lowest level of the Abyss, where the energy used to summon the cards is at its strongest. His shadow double is there pulling cards from the Gacha to keep the supply of cards going after all.
Going back to the "programing" and "serve him like a god" not all of them do that. Most are glad to be granted the chance to be summoned and be of help to him. We've seen so many out following their main mission, but also living as they please. Like the thug group that always has me in giggles when they pop up. You've got merchant ones too. They aren't just made from nothing when they're summoned, but pulled from a parallel world, actually. We don't know if they were still alive there or dead, but we know they come from a different dimension entirely. We've learned this in subtle hints as well, which makes sense since the "Master's" are souls from other worlds reborn with their past life memories in this world.
To also add, not all the females summoned want to screw him or serve him like a god. Treat it more like respecting someone with power [like a child would their parent, authoritative figure, etc.]. As I said before, a handful of them want to have a closer relationship with him, but some just want to have that kinship relationship of a good friend. If this is an issue for you, then you can just ignore everything and just drop the series at this rate. ╮( ̄▽ ̄)╭

"They aren't just made from nothing when they're summoned, but pulled from a parallel world, actually. We don't know if they were still alive there or dead, but we know they come from a different dimension entirely." Ok, is that a theory or canon? Because to be honest, I've been skimming through most of the chapters until I reach the action scenes, because it was getting boring with all the political talk.

Not gonna lie, glad to know there isn't exactly any situations where they just come out and say they want to bang him and I just missed it haha.
I will agree that there is a serious level of cringe to the extent of the devotion, and I'm not entirely sure if it's just the wording they use, or if it truly is that they just worship him (or respect his power?) to that extent. There have definitely been a couple times through the story where some of the summons have expressed their admiration in a way that made me roll my eyes or question their sanity...
Does the harem ruin the story??