He didnt agree, actually we don't know what he said
He literally said it, if you read it properly you see that someone goes like 'true, I used him', then the little guy gets shocked and runs away, but when the other guys were talking about him like that, he just didn't care cause he was used to it.
It's hard not to be biased against the seme since he's acting like such a creeper now.
But yeah, who knows if he said it or if the uke misheard.
And who knows if maybe that was just the start of what the seme was saying. Maybe he had something to say after what the uke heard that gave his statements context.
Either way, with how fast he's trying to fuck, it's hard to believe that the seme wasn't only using the uke back then. At best his main interest is the uke's body (he as much as said so in chapter 2), at worse he is just wanting the uke to only pay attention to him, like he's some pet or doll without free will.
I mean, it seems like you loved him but why did you agree with them when they belittled him? Yaoi logic, sloppily talk shit about the uke when there are others around so their attention won't shift to him and try to steal the uke away, but don't give a shit about the possibility of the uke hearing it.