Thinking that every member of a group is lesser than you is not only extremely stupid but also incredibly dangerous. However, humans are association-making machines and forming slightly negative or positive associations about groups of people is a natural thing, and many of these associations can be true.
One way we learn is by making associations — Barry ate a berry and died; therefore, berry is bad. The strength of an association is based on primarily two factors: on its immediate consequences — has this association served or harmed me? — and how often it is maintained.
The first occurs because not learning something with large consequences makes you unfit to survive. I saw Barry eat a berry and die, but thought it was coincidence; I ate it and died.
The second occurs as a natural byproduct of repeated exposure. Every time I eat an apple, I like it; therefore, I like apples. Every time I eat cheese, I don't like it; therefore, I don't like cheese.
Nobody seems to have any qualms about this idea when it is discussed in the terms I've outlined above, but as soon as we start to introduce how this association works in relation to human groups, there's the inevitable volcanic eruption of emotional rhetoric and blind accusations of racism and bigotry.
The truth is racism and favoritism are just the application of association to groups of people. But there are serious problems with applying associations to people.
Unlike berries, which may have a trait that applies to the entirety of the population — every berry of a given species contains the same poison — human cultures cannot have personality traits that apply to every single member of the population. Racists cannot make this distinction.
Another issue is that the relationship observed may be correlational and not causal. People who watched Barry eat a berry and die may have concluded that the berry was poisonous, but perhaps it was sprayed with a pesticide, and that was the true cause of death. In real life, people often associate a culture with violence and theft when the root issue is economic condition.
Finally, once you believe in an association, you will automatically look for instances that confirm it and dismiss ones that do not. This makes you more confident in your bias even though the pattern may not even exist.
That said, not all associations are without merit.
Since humans have variability, you will not find a personality trait that is constant among every single member of a population. However, what often happens is that a trait will occur a disproportionate amount of the time in one group compared to another. It's well-documented, for example, that people from individualistic countries are more likely to adopt a self-oriented, independent view of the self, while people from collectivist countries are more likely to adopt an inter-dependent, group-oriented view of the self[1]. From personal experience, I vouch for the fact that shame is a much larger part of Lebanese culture than it is of any Western culture.
Would it make sense, then, to say it is 'racist' to say that people from Lebanon are more likely to have a larger group of behaviors that they consider shameful than their Western counterparts, and that people who come from individualistic countries tend to have a more self-centered point of view? I think not. I think this is simply a fact, until the data suggests otherwise.
I think reasonable people will agree with the point I'm trying to make above. It's easy to agree so far because the associations have largely been neutral in nature. Let's try to do something more difficult now, and try to think about slightly negative and positive associations.
I believe that people may, consciously or not, hold associations about groups, and these may be slightly positive or negative in nature. This isn't because people are inherently evil but because our minds are constantly making associations. Ultimately, many people will have associations like this, that they will probably never publicly admit and may not even admit to themselves:
The opposite may occur, as well. Consider, because it is lighter on the spirit, the positive associations one makes toward peoples:
The point is: these statements may be functionally similar to the neutral associations that we made earlier. In essence, they may be true, in a general sense based on one person's interaction with that group. Assuming one is doing everything possible to curtail his biases, if he encounters a group that is constantly associated with a certain behavior, whether positive or negative, is he wrong to assume that that members of that group may exhibit it in the future? I don't think so, and I don't think this is where the majority of the problem of racism lies. In any case, making small associations about groups of people does not pose a significant threat to anybody and can even be accurate in some cases.
Associations can be weakened or strengthened over time, however, and one must be cognizant of the spirit of one's associations and why one has them. Blind hate may start off as an 'innocent bias.' Alternatively, one can start with a weak association and have it extinguished over more exposure to the group. We are constantly learning (whether correctly or incorrectly is another matter!)
The real danger comes from having a strongly negative association that one applies indiscriminately to an entire population.
Unlike the associations above, strongly negative ones carry very significant risk. Once an association is extremely negative, the people subjected to it are typically perceived as subhuman, inferior, or, just generally, 'enemies.' Pair a strong negative association with indiscriminate application to an entire population and you have the recipe for mass murder and other such atrocities.
The issue with the indiscriminate application of such views is that innocents will always be harmed. That is why, even amongst groups I find deeply abhorrent — like Nazis and Israelis — it's wrong to wish harm on all. The guilty and the innocent are never perfectly separable, and treating them as if they are is the very error that makes atrocities possible.
To recap:
You are not some evil monster for doing what your brain naturally does. Having slightly negative or positive associations to a group of people is natural and does not harm or benefit anybody much, but be suspicious of the associations that you have, as they may be completely untrue and pay particular attention to the negative biases you have to prevent them from going overboard.
[1] Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98(2), 224–253.