Does the harem ruin the story??

Depends on how you view a harem. There doesn't seem to be any romantic feelings from the main male character for any of his female subordinates. He only seems to view them as valued comrades and friends. They seem to worship him to an extent, but not sure if it is romantic or more of a a diety sort of worship, as in he is the greatest and there is none better.
╮(￣▽￣)╭ basically, I'm not getting any romance vibes out of this harem, so I don't really see it ruining the story in that way.