In January 2020, Detroit police drove to Robert Williams’ house and arrested him on his front lawn, in front of his wife and his two young daughters. He spent about thirty hours in a cell. The case against him was that a facial recognition system had matched a grainy still from a store’s surveillance video, taken during the theft of several watches from a Shinola shop, to the photo on his expired driver’s license. The match was wrong. Williams was not the man in the footage and was nowhere near the store. He is believed to be the first person in the United States wrongfully arrested because of a face recognition error [1].

That case is a good place to start a series on AI ethics, because almost nothing about it involves a villain. No engineer set out to arrest the wrong Black man. The camera, the matching algorithm, the officers, the database: each piece did roughly what it was built to do. The harm came out of the system as a whole. This first part is about where that kind of bias actually comes from, why “just remove the bias” is a much harder instruction than it sounds, and why some of it turns out to be mathematically impossible to remove without giving up something else you also wanted.

Bias enters through the data, the labels, and the goal

Around 2014, Amazon started building an experimental tool to screen resumes. The idea was ordinary: feed the machine a decade of past applications and the hiring outcomes attached to them, and let it learn to spot promising candidates. By 2015 the team noticed the model was penalizing women. It downgraded resumes that contained the word “women’s,” as in “women’s chess club captain,” and it marked down graduates of at least two all-women’s colleges. Amazon tried to correct for the specific terms, could not convince itself the tool was clean, and eventually scrapped the project [2].

Nothing in that system was told to prefer men. The bias came from the training data. Most resumes Amazon had received over the previous ten years came from men, because tech skews male, so the patterns the model learned to reward were the patterns that described the men who had been hired. The machine did exactly what it was asked: reproduce the past. The past was skewed, so the future it proposed was skewed too.

This is the first thing to hold onto. A model has no opinions. It has a training set, a set of labels telling it which examples count as good, and an objective it is trying to optimize. Bias can enter at each of those three points.

It enters through the data when the examples are not representative, as with Amazon’s mostly-male resumes. It enters through the labels when the human judgments the model learns from carry human prejudice: if past loan officers or past managers made biased decisions, a model trained to imitate them inherits the bias, laundered into something that now looks objective. And it enters through the objective, the single number the system is built to maximize. A model tuned purely to predict “who did we hire before” will happily learn “hire people like the ones we hired before,” which is not the same as “hire the people who would do the job best.” The goal you write down is rarely the goal you actually have, and the gap is where a lot of bias lives.

Why “just debias it” is so hard

Amazon’s first instinct was the obvious one: find the offending signal and delete it. Strip out the word “women’s.” Do not let the model see gender. This almost never works, and it is worth understanding why, because the same failure recurs everywhere.

The reason is proxy variables. Even after you remove the sensitive attribute itself, other features quietly stand in for it. Gender was not really encoded in the token “women’s.” It was smeared across the whole resume: the sports played, the phrasing, the choice of verbs, the schools attended. Remove one proxy and the model leans on the others. This is the machine-learning version of a much older problem. Mortgage redlining did not need a box marking race, because a postal code did the job. Correlated features carry the forbidden information around the fence you built.

You can watch this play out in lending, where the stakes are money and the data is good. A study by Bartlett, Morse, Stanton, and Wallace at Berkeley examined millions of US mortgages and found that Latino and Black borrowers were charged noticeably more for the same loans: about 7.9 basis points more on purchase mortgages and 3.6 on refinances, which they estimated at around 765 million dollars a year in extra interest [3]. The interesting part for our purposes is what happened with the algorithmic lenders. Fintech underwriting, with no loan officer in the room and no face to react to, did better. It cut the pricing gap by more than a third and showed no discrimination in who got rejected. But it did not reach zero. The algorithms, trained on market data shaped by the same history, still charged minority borrowers more [3]. Taking the human out of the loop helped. It did not make the problem disappear, because the bias was never only in the human. It was in the data the human generated.

When fairness definitions collide

Here is where the topic stops being a matter of trying harder. In 2016, ProPublica published an investigation into COMPAS, a risk score used across US courts to estimate how likely a defendant is to reoffend. Looking at more than 7,000 people arrested in Broward County, Florida, the reporters found that among defendants who did not go on to reoffend, Black defendants were flagged as high risk almost twice as often as white ones: a false positive rate of about 45 percent versus 23 percent. White defendants who did reoffend were more often mislabeled as low risk [4].

Northpointe, the company behind COMPAS, pushed back with a claim that also turned out to be true: their score was calibrated. A given score meant the same probability of reoffending regardless of race. A “7” carried the same real-world risk for a Black defendant as for a white one [4]. So one side said the tool was biased and the other said it was fair, and the strange thing is that both were right. They were measuring fairness two different ways.

This is not a debate you can win by arguing harder, and two papers proved it. Kleinberg, Mullainathan, and Raghavan showed that three natural conditions you would want from a fair risk score cannot all hold at once, except in special cases that essentially never occur in the real world, such as the groups having identical base rates or the predictions being perfect [5]. Chouldechova, working directly on the COMPAS dispute, showed the same tension in plain terms: when the underlying rate of the outcome differs between two groups, a score that is calibrated cannot also have equal false positive and false negative rates across those groups [6]. Pick calibration and you are stuck with unequal error rates. Equalize the error rates and you break calibration. You cannot have both.

That is what “fairness impossibility” means, and it is the single most important idea in this article. Fairness is not one property you can turn up. It is several properties that pull against each other, and once the base rates differ, satisfying all of them is not merely difficult but ruled out. COMPAS was not a case of a company being lazy. It was a case of two incompatible definitions of fair, and a design choice about which one to honor that nobody had made explicitly. The math does not tell you which definition to pick. That is a value judgment, and pretending the algorithm made it for you is how the value judgment gets hidden.

Uneven error rates are not evenly distributed

The abstract point about error rates has a very physical face on it. In 2018, Joy Buolamwini and Timnit Gebru tested three commercial gender-classification systems and reported the results by skin tone and sex. On lighter-skinned men the systems were nearly perfect, with error rates under 1 percent. On darker-skinned women they failed up to about 35 percent of the time [7]. The average accuracy looked fine. The average hid the fact that the failures were piled almost entirely onto one group.

This was not one bad vendor. In 2019 the US National Institute of Standards and Technology ran the largest study of its kind, testing 189 face recognition algorithms from 99 developers. For one-to-one matching, the kind used to unlock a phone or check a document, false positive rates for Asian and African American faces ran from 10 to 100 times higher than for white faces, depending on the algorithm. For the one-to-many searches used by police to find a suspect in a database, the highest false positive rates fell on African American women [8]. A false positive in a phone unlock is a nuisance. A false positive in a police database is Robert Williams on his front lawn.

Trace it back and the shape is the same as everything above. The systems were trained on faces, and the faces they saw most were the faces they got best at. The objective rewarded overall accuracy, and overall accuracy is happy to be excellent on the majority and poor on a minority, because the majority dominates the average. Nobody encoded a preference for lighter skin. The pipeline produced one anyway, and then a police department wired that pipeline to the power of arrest.

Treating people the same is not the same as affecting them equally

The law has been wrestling with this longer than computer science has, and it drew a line worth borrowing. In 1971 the US Supreme Court decided Griggs v. Duke Power. The company required a high school diploma and a passing score on general intelligence tests for its better-paying jobs. On paper the rule applied to everyone. In practice it screened out Black applicants at much higher rates, and neither requirement had been shown to predict who could actually do the work. The Court ruled 8 to 0 that Title VII bans practices “fair in form, but discriminatory in operation,” and that “good intent or absence of discriminatory intent” does not save a practice that operates as a “built-in headwind” for a protected group unless it is genuinely related to the job [9].

That gives us two distinct ideas. Disparate treatment is deciding differently because of who someone is. Disparate impact is a neutral rule that lands unequally, whatever the intent behind it. Almost every case in this article is the second kind. Amazon’s tool did not have a rule against women; it had an impact against them. The face recognition systems held no animus; they had error rates that fell unevenly. Intent is the wrong thing to look for. A model has no intent, and Griggs already told us intent was never the point.

The doctrine also shows why the fix is genuinely hard, not just neglected. Consider the Apple Card. In 2019 it was publicly accused of sexism after several people, including a well-known software developer, reported that men were offered far higher credit limits than their wives despite shared finances and better credit scores. New York’s Department of Financial Services investigated, analyzed the underwriting of roughly 400,000 applicants in the state, and found no violation of fair lending law [10]. That result deserves to sit next to the others, not because it clears algorithmic lending, but because it shows the limit of the tools. Proving disparate impact requires the right data, the right comparison, and a standard the evidence can actually meet. Sometimes the impact is real and provable, as in the mortgage study. Sometimes an investigation with far more access than any outsider has still cannot establish it either way. The honest position is not that every disputed system is guilty. It is that “we treated everyone the same” is not, and since Griggs has never been, a sufficient defense.

What this leaves us with

The through-line is that bias in these systems is not usually a mistake someone made and can un-make. It is a property of the pipeline. It enters through data that records an unequal past, through labels that carry human judgment, and through an objective that optimizes an average and is indifferent to who absorbs the errors. Some of it can be measured and reduced, as the fintech lenders reduced it. Some of it, once base rates differ between groups, cannot be fully removed under every definition of fair at the same time, because those definitions are mathematically incompatible and someone has to choose among them.

None of that is an argument for throwing the systems out, and none of it is an argument for trusting them. It is an argument for a specific kind of honesty: about what the training data actually contains, about which fairness definition a system was built to satisfy and which it therefore sacrifices, and about the difference between treating people identically and affecting them equally. The next part turns from where bias comes from to what we can actually do about it, and where the current toolkit runs out.

References

1. American Civil Liberties Union (2021). Williams v. City of Detroit. ACLU case page. https://www.aclu.org/cases/williams-v-city-of-detroit-face-recognition-false-arrest
2. Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
3. Bartlett, R., Morse, A., Stanton, R., & Wallace, N. (2019). Consumer-Lending Discrimination in the FinTech Era. NBER Working Paper No. 25943. https://www.nber.org/system/files/working_papers/w25943/w25943.pdf
4. Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine Bias. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
5. Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores. arXiv:1609.05807. https://arxiv.org/abs/1609.05807
6. Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data, 5(2), 153-163. https://arxiv.org/pdf/1703.00056
7. Buolamwini, J., & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of Machine Learning Research (Conference on Fairness, Accountability, and Transparency). https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212
8. Grother, P., Ngan, M., & Hanaoka, K. (2019). Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects. NISTIR 8280. National Institute of Standards and Technology. https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software
9. Griggs v. Duke Power Co., 401 U.S. 424 (1971). U.S. Supreme Court. https://caselaw.findlaw.com/court/us-supreme-court/401/424.html
10. New York State Department of Financial Services (2021). Report on the Apple Card Investigation. https://www.dfs.ny.gov/reports_and_publications/press_releases/pr202103231