We can rule out his dad because the mom said she could never face his father if something happened again (which is typical phrasing used by the surviving parent when the other one is dead, and on rarer occasions when they are divorced). If the dad had done something then she wouldn't have given a sh*t about what he thought. The stepdad can be ruled out since they don't meet until the present.
The way the flashbacks are presented it was most likely a stranger, or possibly a neighbor/ teacher, or maybe a coworker/friend of the mom's. Though I lean strongly towards it was a stranger who kidnapped him, which is why they never delve farther in it because it wouldn't be necessary to develop his character further.
Didn’t his friend interrupt as he said if only he got back earlier. http://iweb9.mangapicgallery.com/imgfiles/kousoku_syndrome/2/017.png it seems like he blames himself. Could it be someone mutual to them both at the time? I know he’s doing better because of otomo but I’m still concerned about the underlying trauma, it looked like he was really young and his mum and friend keeps blaming themselves still. (I’m strangely concerned over a fictional character )

are we not gonna talk about the guy who caused his trauma with men? no? okay.