You may just have missed the point of this story completely. Fumichi didn't care about any of it. He was like a doll. Empty. Sazawa says so when they are bathing. The rape part just illustrated that. Fumichi didn't care about it. He just regarded that as routine. Yeah sure in some level he was aware of it and didn't like it. But his self awareness and consciousness was too numb to give the issue the weight it's supposed to have in Fumichi's mind.
In Fumichi's mind is just like a rock in the shoe. Annoying and mundane. He doesn't think more to it even if he should.
And this story isn't about rape but about Fumichi growing up, and caring about himself, about others.
So that's why the matter of rape isn't addressed further. It's dealt with exactly in the right way to illustrate the state of mind of Fumichi

Fumichi's past wasn't clarified. Why his customers treated him badly wasn't really regarded either.