Why do some people think that if a black man is a high achiever, it must be because of affirmative action?

I've noticed that there are a lot of people here who seem to think that Barack Obama is where he is, thanks solely to affirmative action. Why is it so hard for some to believe that perhaps he is where he is today, because he actually WORKED for what he has? If he were white, would people be just as skeptical about his accomplishments?