
He never explained why he said "I don't wanna!"

His Dad didn't die until his second year in college, which is when he broke up with Masaki and dropped out of school. He tried to get his contact info two years earlier at their graduation.
https://www.mangago.zone/read-manga/ore_no_oniku_chan/mdx/d_chapter-5/5/
he picked her up with one hAAAAAAAAAND