Generic Anime Caracter's feed

Why are so many people freaking out about mistresses that aren't even official concubines??? They're just potential candidates that he took in so the nobles would leave him alone and because his wife pissed him off saying he should do it (when he didn't even want to in there first place). He'll, he doesn't even really interact with any of them because he really loves his empress, he's the only one we see him dance with. And there aren't even rumors of him being interested in or paying any attention to any of the girls that came to the palace.