People can be "not gay" and still be romantically attracted to the same sex.
"How is that not gay?" Ask or see for yourself if you'd feel the same if it was with other people. If you can easily see yourself with others of the same sex, then maybe. If not, if you only feel this way towards 1 particular same sex person, then it's different.
In those cases, it's the person you fell in love with, not their sex.
I like to put it as "falling in love with their soul". Our souls are neither male nor female. It's pure being.
I read this once before, "falling in love is your soul recognizing it's counterpart in another person". That's simply what's happening.
You can't be attracted to a man as another man and be straight. Perhaps he's pansexual (don't care about gender, like you said) or demisexual. It's okay to simply not want to label yourself as gay, but it's definitely not heterosexual.
Bitch if he's not gay, why did he confess to you!?