AI bias occurs when an AI system produces unfair or skewed results because of flawed data, biased assumptions, or problematic design choices. Just like humans can have unconscious biases, AI systems can inherit and even amplify biases from their training data and creators.
Understanding AI bias is crucial because AI systems are increasingly making decisions that affect people's lives - from job applications to loan approvals to criminal justice.
When AI systems are biased, they can:
When training data doesn't represent all groups equally. For example, if a facial recognition system is trained mostly on light-skinned faces, it will perform poorly on darker skin tones.
When the data used to train the AI doesn't reflect the real world. If you train a hiring AI only on resumes of people who were hired in the past, it may discriminate against qualified candidates who don't fit historical patterns.
When AI learns from historical data that reflects past discrimination. For example, if men were historically hired more often for tech jobs, an AI might incorrectly learn that men are better suited for these roles.
When developers design AI to confirm their existing beliefs rather than discovering objective truths. This can happen when choosing which features to include or how to interpret results.
When the way an algorithm processes data creates unfair outcomes. Even with good data, poorly designed algorithms can produce biased results.
When human labelers unknowingly add their own biases while tagging training data. For example, labeling certain behaviors as "suspicious" based on stereotypes.
AI systems used to predict recidivism (re-offending) have been found to incorrectly flag Black defendants as higher risk twice as often as white defendants, even when controlling for prior criminal history.
Amazon had to scrap an AI recruiting tool because it discriminated against women. The system was trained on resumes submitted over 10 years, which were predominantly from men, so it learned to penalize resumes containing the word "women's."
Studies have shown that commercial facial recognition systems have higher error rates for people with darker skin tones and for women, with the highest error rates for dark-skinned women (up to 35% error rate vs. less than 1% for light-skinned men).
An algorithm used to determine which patients need extra medical care was found to favor white patients over Black patients. The bias occurred because the system used healthcare spending as a proxy for health needs, but Black patients historically have less access to healthcare.
Try the Bias Detective game to identify different types of bias in real scenarios!