It would make sense, in the cliche way that is. I wanna say that he had a traumatic past involving a summoner not being able to control the magical beast, or a summoner who purposefully used the magical beast for evil deeds. I'm gonna go with the latter and I'm also going to guess that the man in the white cloak is involved in some way.
Or maybe it can be both... it's a little out there but think about this.
What if a sorcerer, a wizard, a mage, whatever you want to call them, comes out of nowhere and starts spreading all types of panic. It could be directly or indirectly or even both. Summoners come and try to help fight them/him off, only for the sorcerer to take control of their magical beasts and turn them against their summoners.
I know, very far fetched, but interesting none the less. Or what if the concept of an out of control magical beast is what truly scaring him? No, I doubt that.
Who wants to bet that the reason red head dude hates summoners has something to do with her trash ex that used her to pretend to be a summoner?