In context he is a crybaby so tears were expected. And he was scared that he was smiling and laughing, they both know that isn’t something he ever does. He’s known him since they were 5 and he’s only seen him do that once. He actually even says “yes” on one of the panels when they’re having sex. Implying he’s enjoying it. He might’ve been frazzled but didn’t want him to stop.
https://www.mangago.zone/read-manga/given/mf/br_chapter-18949/pg-20/
Hiiragi say himself that his face scared him
He said that because he was smiling. It’s in the context. That is said after the panels of him showing his teeth in a smile, which he hasn’t seen in 10 years, the procession of the panels let’s us know it’s not the sex he’s scared of but his smile. He also says “yes” in the panels where they’re having sex.

I just don’t understand how you would have the heart to continue doing something that hurts the one you say you love
Clearly Hiiragi is distraught and crying, how and why could he not stop? Especially since the tears seem to come from fear and distress, not pleasure
Whether Hiiragi said he was ready at the beginning doesn’t mean everything is allowed, and that stopping isn’t possible.