I thought Numbers were a pretty well known group

They are, but only by mouth of others. People outside where they trained never knew how strong they are, most of those who met them once never alive to tell the tales. So ppl who are after them only think that the numbers are “mildly” stronger than the rest