Besides what you said, the girls around him are far too young for him and he recognizes that. Perhaps, to him it would be something disgusting like paedophilia or something. I remember that in one chapter he did show some interest, but in a woman much older than his current age (around 40 or something?)
But still, the blond (long haired one, Milly?) she is so freaking disappointing. Altho I am glad the protago isn't a pervert, the girls around him besides the booby girl and his master in the previous life are all so..frustrating. That Milly being the leader is the most annoying of it all. If the protago isn't there, they will head on die in a matter of minutes..if not seconds.

all the old men isekaid ive read are not even self conscious,
Its not that I want them to be always perfect but at least try not to make them act like awkward teenagers unaware of the opposite sex.
at least he got it right with him not getting flustered with half naked girls since maybe in his past life he coulda fked some and is desensitized by it.