I'm guessing Togawa (as shown in ch 4 already) didn't allow Nozue to give a response because he thought he would definitely be rejected. Instead he closed things off by asking if they could continue their superior-subordinate relationship after which I guess that Nozue left by a brief goodbye without fully responding to Togawa.
But but but, just because Togawa gave him a graceful exit, doesn’t mean he has to actually go through it...
I'm guessing he felt flustered and in the heat of the moment felt like running away from his feelings. Plus there's not much he can say when Togawa begs him to keep things as they are
Noooooooooo... did he leave poor Togawa hanging after that confession?!???