I get what you mean. I feel like there was something missing in this manga, because you're right there were a couple of paths that it seemed to go down but eventually just sort of fizzled out. Maybe the mangaka wanted to have false roads or maybe just isn't good writing but I felt that this one had a lot of potential but was kind of a let down in the end.
It wasn’t “we used to have sex before you lost your memory”, it was just aachan daydreaming, his way of punishing himself for ryo falling. The ryo in his dreams still remembered what had happened, and acted sadistically to punish aachan. And as to why they were having sex.. well, aachan liked ryo, so?
yeah I feel the story tackled many heavy psychological issues but lacked depth in resolution. It would have been nice if they had stretched the story a bit because really, such serious issue don't get resolved in just a few days. Take Sora's case for example. It's true it varies from person to person but to myself as a reader, I feel unsatisfied. But, well, the main couple is happy. Let's hope Sora finds his own too.
Now maybe I just didn't understand but I'm pretty sure this was one big plot hole. Why was there a set up of "we used to have sex before you lost your memory" and then later a set up of "amnesia boy is actually a secret badass either hiding his powers or who forgot he was a badass" like this just left me confused from binning to end