
The term "Colorism" does not even officially exist. It's not really a word with an official definition yet. With that said, although it happens often in the context of people among the same race, it is not exclusive to the banner of racism. It's simply a culture's preferential tendency to lean to a certain type of skin color. Its a form of prejudice, but not always racism, in fact it would seem to contradict the very definition of racism. Colorism usually happens in the same race and usually has nothing to do with discriminating from one's race, but rather it focus on their skin color preference.
For reference this is the official definition of Racism from Webster
"a belief that race is the primary determinant of human traits and capacities and that racial differences produce an inherent superiority of a particular race"

I believe you are thinking of "Race" in a biologic sense.
'Racism' was actually a termed coined with the idea of classifying humans to groups based off their shared characteristics, so it's not a misused word; that was the intent when it was made.
Race defined as human categorization according to shared characteristics etc is accepted in Anthropology as it's official definition. Among Biologist, it may carry a different meaning.
Race has different meanings. It could be a running competition, a taxonomy, or a reference to a group of humans sharing specific characteristics.
Therefore, it's not a misused word, you are simply looking at a different definition of it.
This manga doesn't have racism. Racism is discrimination against someone because of their Race and not their skin or hair color. A person with African descent who is light skinned can still face racism for being "black" and that refers to their descent, not per say their skin color.
There is discrimination against a person's look here, but it's not towards their race. It is important to note that skin color is NOT exclusive to any race.
To put it in isekai perspective, their racism usually is when humans discriminate against beastkins, dragonkins, humanoid etc.