Maybe bec. the story somehow is creating a line of admirers for the mc. A lot of story out there started their harem with admirers and somehow in the end they tend to entertain and include them in a romantic relationship just bec. They deserve it or the mc doesn't want to hurt their feelings so it turns to be a harem. However this story seems more focus on the mc it hardly show any kind of affectionate moment between him and those girls. So even if this will turn out to be harem I think we will not be able to see a generic harem that's focus on romance, this is more of his journey and him alone.
I agree I don’t think the focus will be on his romantic relationships if he does end up with a harem. Especially since he doesn’t seem to have much interest in his relationships now and how the focus now isn’t being put on those.
For those who are confused about whether or not this has harem, here is a more in-depth explanation. The ML has one concubine and two female admirers, which are barely mentioned, shown, and have little significance to the story. The ML also shows no romantic interest in any of them. If the harem aspect is making you question if you should read this or not, don't let it stop you from at least trying it out. After reading to chapter 91 (the amount posted when I wrote this) I’ve seen other comments and topics about the harem, and personally I’m just confused. I don’t understand how having one wife is considered harem, and personally while reading this I never thought about if it was harem or not, it just never came to my mind. Furthermore, give it a try. （￣へ￣）