If i remember correctly he once said that he feels shy/embarrassed to take it off because of his quite slim body (=・ω・=)
He feels insecure because his body is thin. So, he feels more comfortable when he is dressed."
Ep 46. It's very briefly gone trough but it's quite clear deal after tht :D Taka just is more comfy with his clothes on
+ it's not like Yu cares about it so it's not really something that "needs fixing"
All 152 pages i notice that the top doesn’t take his shirt off AT ALL ( except for the one time bath scene)
I hope later on in the story it will explain that