Each culture has its own unique worldview, values, and practices. Humans tend to be suspicious of worldviews/values/practices that differ from their own. In some cases, we can even despise all or some aspect of certain cultures (often for illegitimate reasons such as “I had an experience in which a person of X race did me wrong, therefore I don’t like people of X race”). Many times, the skin color of the people in the culture we despise differs from our own as well. But is the color of their skin the cause of the animosity? No, I don’t think so. The person from culture A with skin color B despises people from culture X with skin color Y, not because he hates skin color Y, but because skin color Y serves to identify the people who belong to the culture who thinks/acts in ways he despises. In other words, race is incidental to the animosity, not the source.
Think about it. When someone says, “I hate Chinese people,” and you ask them why, you will often be given a list of behaviors s/he believes are typical of Chinese people—behaviors s/he does not like. It is the behavior/culture—not the skin color—that is being despised. Indeed, if someone of their same race did the same things, s/he would despise that person just as s/he despises the Chinese (if s/he did not feel the same way about someone of his/her own race who behaves the same way, then that would be good reason to think s/he is truly racist). Indeed, you often find animosity between sub-groups of the same race for either cultural or political reasons (e.g. French-German animosity), and yet no one labels it racism. But if the French had dark skin, people would call it racism. This tells me that our definition of racism is misguided. Racism is animosity toward others primarily due to the color of their skin or their ethnicity.
Of course, no matter what we call it, and no matter what motivates the hate, it is wrong.