It's not a harem. The chivalrous elf lady is the only female love interest for the MC. The others just gush over their romantic moments. I do agree with the second complaint though.
Ya'll do know that he doesn't actually need to be with all of them for it to count as a harem? If it's one guy surrounded by a bunch of women, its a harem. If its one woman surrounded by a bunch of men, its a REVERSE Harem. Regardless if he's with one romantically, its a harem.
*SPOILER*
Sorry to disappoint but he has 3 or 4 wives
and three daughters. Three of his wives has one daughter each.
Haruka (Maria's Daughter)
Etoile (Sofia's Daughter)
Flora (Mani's Daughter)
literally what the fuck