Well, that's because it's a lot easier for one's ex lover to notice a missing, non-returned expensive brandname shirt (of the type the seme here probably wears), than it is for them to notice (or even get angry over) a missing toothbrush that one failed to give back to them...
And while no ex lover would have any right to get mad at you for throwing away something so trivial like their old toothbrush (rather than going through the trouble of sending it back to them) they might have more grounds to get angry over you "throwing away" a shirt, rather than seding it back, because shirts can be costly.
I mean, the mc isn't rich like the seme, so he is more economically minded and waste-avoiding. That's why he's afraid the seme might notice if went and kept one of his shirts, so he chose to keep one of his belongings that he knew no "nornal" person would miss or get angry over (if their ex suddenly claimed they'd thrown it away...)
Can’t believe out of everything he could’ve kept, he held on to a nasty used toothbrush?? Like sir not a shirt that smelled like him but the thing deep throating his morning breath?? Anyways dropping this for a bit and coming back for the last 10 chapters