I guess my memory is shit because I don't remember him taking that girl as concubine, I do...

I guess my memory is shit because I don't remember him taking that girl as concubine, I do remember her father saying he should take responsibility but the girl he's with (sorry I'm really bad at names) gets really upset about him having a concubine so he doesn't want it and the yin yang girl doesn't want either so I was thinking she isn't his concubine in the end, am I crazy? Because I remember it like this in the novel (it's been a while since I stopped reading it) but everyone is talking about harem in the comments and now I'm questioning myself （/TДT)/