I honestly believe it's because the audience is young. This generation has become far too eager to label every single intercation between people with any form of an age gap as pedophilic or grooming.
Child lover/ Pedo/ Groomer... its like they heard these words once and just ran with it to the moon and back. I once saw a post about a guy (18) who kissed another (16) and the amount of people harrasing him and calling him a pedophile over a 2 year age difference was insane. I'm beginning to worry about how casually people have been using these terms.
Anyways as you've said the themes in this story are very tame cause nothing happened until the seme was a grown ass man and he was the one pushing for a relationship the entire time. How anyone could call the uke a groomer is being comprehension (and once again proves that they just be throwing these terms around without fully understanding their meaning).
Tl;dr: The past between the characters looks very tame if you've been reading east Asian works for a long time and/or you can dissociate a fictional story from real life situation.
Longer version:
I have been reading mangas for many many years and I have read several mangas with similar themes to this one. It never occurred to me that this might be problematic. I think it's very tame compared to some other prominent older works (eg. Hana to Akuma, Usagi Drop). Am I too old or the people in the comments too young? Lol
It's interesting how people read stories today. Until a few years ago I didn't see so many people complaining about potential crimes in a fictional story. (eg. BJ Alex was very well received when it first came out but I see people pointing out the problematic parts in the story now.)
I am into true crime so I am familiar with things like grooming though in my country it's not discussed often. Not that it doesn't happen but it's not something people would immediately think of when they see a fictional character in a fantasy setting taking care of a dragon in a human form only to end up with the said dragon human in the future. (it's better than taking in a baby, raising them and eventually marrying and having kids with them because you're an immortal vampire and age is just a number lol).
I think it's also cultural. There are so many Chinese, Japanese and Korean stories with similar themes in a fantasy story and they keep increasing in number. (All the shizuns would be labelled predators irl lol) Probably because the readers of the native work don't find it as problematic as the international audience since if the authors saw the complaints they'd probably change the way they write the stories (over time works like Hidoku shinaide, Dakaretai Otoko have shifted from the initial "I assault you because I like you" to more acceptable relationship between the characters and I think more wholesome stories like Sasaki to Miyano are becoming popular. Even in Chinese danmei, most popular stories now don't start with an assault at least. Can't say the same about "grooming" though).
I don't think people who think of the development between the characters in this story as grooming are absolutely wrong. But I do wonder what is the demographic that think of it this way. Is it the people who have recently started reading BL and/or are also very young (under 18) or people who have become too sick of this kind of development over the years XD