In 2019, a team led by Ziad Obermeyer looked inside a commercial algorithm used by hospitals and insurers to decide which patients needed extra medical attention. Tools of the kind they studied touched the care of roughly 200 million people in the United States every year. The algorithm produced a risk score, and the higher your score, the more likely you were to be flagged for a high-touch care program. It worked, in the sense that it ran, it produced numbers, and clinicians used them.

It was also racially biased in a way nobody had noticed, because the score was built on a quiet substitution. The model did not predict how sick you were. It predicted how much you would cost. Because less money had historically been spent on Black patients with the same conditions, they had to be considerably sicker than white patients to earn the same score. Obermeyer’s team found that correcting the target, predicting illness instead of spending, would have raised the share of Black patients flagged for additional help from 17.7% to 46.5%. [1]

The bias was not hidden in the sense of being encrypted or secret. It was hidden in the sense that the system offered no way to see it. You put a patient in, you got a number out, and the number came with no account of itself. That is the black-box problem, and it is the subject of this piece: what it means to demand that a model explain itself, whether the explanations we get are worth anything, and who is on the hook when the answer is no.

The box and what is inside it

A modern machine learning model is a function with millions or billions of tuned parameters. A deep neural network trained on medical records does not store a rule like “if cholesterol is high and the patient smokes, raise risk.” It stores a vast web of weights that, taken together, produce an output. No single weight means anything on its own. There is no line you can point to and say: this is where the decision happened.

This is not a temporary state of ignorance that better engineering will clear up. It is a property of how the models are built. We trade transparency for performance. The same flexibility that lets a network pick up on subtle patterns in data is exactly what makes it resistant to being read back out in human terms. For a movie recommendation, nobody cares. For a decision about bail, a mortgage, a cancer screening, or which patients get scarce clinical attention, the inability to say why starts to look less like a technical footnote and more like a governance failure.

Two different responses have grown up around this problem, and keeping them apart is the whole game.

Explaining a box versus building a glass one

The first response is called explainability, or post-hoc explanation. You keep the black box and bolt an interpreter onto the outside of it. The interpreter watches inputs go in and outputs come out and tries to reconstruct a human-readable story about what the model is doing.

The two best-known tools here are LIME and SHAP. LIME, introduced by Marco Ribeiro and colleagues in 2016, approximates the complicated model in the small neighborhood around one prediction with a simple linear model, on the theory that even a wildly nonlinear function looks roughly straight if you zoom in far enough. [2] SHAP, from Scott Lundberg and Su-In Lee in 2017, borrows an idea from cooperative game theory: it treats each feature as a player and computes how much that feature contributed to pushing the prediction up or down, using a quantity called the Shapley value. [3] Both give you the same kind of artifact: a little bar chart saying this loan was denied 40% because of income, 30% because of credit history, and so on.

The second response is interpretability: do not build a black box in the first place. Build a model whose workings are legible by construction. A short decision tree, a scoring system with a handful of weighted factors, a sparse linear model. You can read the whole thing. There is nothing to reconstruct because nothing was hidden.

These sound like two routes to the same destination. Cynthia Rudin, a computer scientist at Duke, has spent years arguing that they are not, and that we routinely confuse them to our cost.

Why the explanation is not the model

Rudin’s 2019 paper in Nature Machine Intelligence has a title that doubles as its thesis: “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.” [4] Her central point is easy to miss because it sounds pedantic until you sit with it. A post-hoc explanation is a model of a model. It is a second system, trained to approximate the first. If it reproduced the original perfectly, it would just be the original. So by definition it is wrong somewhere. The explanation and the model agree most of the time and diverge some of the time, and you generally cannot tell which case you are looking at.

That gap is not academic. In 2020, Dylan Slack and colleagues showed you can weaponize it. They built classifiers that were blatantly discriminatory on real inputs but could detect the artificial, perturbed data points that LIME and SHAP use to probe a model. When the explainer came sniffing around, the model behaved itself and produced an innocent-looking explanation citing harmless features. On actual decisions, it went on discriminating. [5] The explanation was not just imperfect. It was an alibi, and it was cheap to manufacture.

Rudin’s further claim, backed by a growing body of work, is that the supposed price of interpretability is often imaginary. On a lot of real-world problems with structured, meaningful data, a well-designed interpretable model performs about as well as the black box it would replace. The recidivism-prediction tool COMPAS, at the center of a famous fairness controversy, is a proprietary black box using upward of a hundred variables. Rudin and others have shown that transparent models built on just a handful of variables predict reoffending roughly as accurately. If a model you can read on an index card matches a model nobody can read, the black box is not buying you accuracy. It is buying you deniability.

None of this means post-hoc tools are worthless. For a low-stakes system, or as a debugging aid for the engineers who built the thing, LIME and SHAP are genuinely useful. The argument is narrower and sharper: when the decision is consequential and irreversible for the person on the receiving end, an approximate story about an unreadable model is the wrong tool, and reaching for it lets everyone feel accountable without anyone being accountable.

Who pays when the box is wrong

Suppose the model does cause harm. A qualified applicant is denied a loan, a patient is triaged away from care they needed, someone is flagged as a fraud risk and frozen out of their account. Who is liable?

The honest answer, in most places today, is that it is complicated, and the complication runs in favor of whoever deployed the system. To win a negligence claim you typically have to show what went wrong and how the defendant’s conduct caused your loss. With a black box, the evidence you would need to do that is locked inside a system you cannot inspect, often owned by a company that treats it as a trade secret. The opacity that makes the model hard to govern also makes it hard to sue over. The burden of proof sits on the party with the least access to the proof.

Europe tried to close this gap and then backed away, which tells you how hard the problem is. In September 2022 the European Commission proposed an AI Liability Directive, designed to ease the burden of proof for people harmed by AI systems, partly by letting courts order disclosure of evidence about high-risk systems and by presuming a causal link in certain cases where a provider had broken the rules. [6] It never passed. On 11 February 2025, the Commission listed the proposal for withdrawal in its work programme, citing no foreseeable agreement among the member states and a broader push to simplify digital regulation. [6] The withdrawal became official later that year.

So the dedicated liability regime is, for now, gone, and the ground it was meant to cover is held by a patchwork: general product liability rules, the revised Product Liability Directive, sector-specific law, and national tort systems that were not written with adaptive statistical models in mind. The result is that the accountability question the black box raises most sharply, who answers for the output, is the question the law is least ready for.

When the box picks a target

The examples so far are civil: a loan, a triage score, a frozen account. The accountability problem does not stay civil. On 28 February 2026, during the opening of the war between the United States and Iran, a US Tomahawk cruise missile destroyed the Shajareh Tayyebeh elementary school in Minab, in southern Iran. More than 150 people were killed, including over a hundred children, along with teachers and parents. Early counts ranged from roughly 156 to 168 dead depending on the source. [12]

The strike did not come from an autonomous weapon that picked its own victims. It came from a targeting pipeline with a black box inside it. US forces used Palantir’s Maven Smart System, which embeds Anthropic’s Claude to help rank potential targets by strategic importance and to work through the volume of intelligence behind each strike. [12] A human approved the strike, which is the safeguard everyone points to. Anthropic’s chief executive later said the principle “that a human makes the final decision” had been followed, while also admitting the company did “not know exactly how” its models had been used in the operation. [13]

The proximate cause was mundane and human. US Central Command built the targeting coordinates from intelligence the Defense Intelligence Agency had not updated to reflect that the site was now a school. Former officials were blunt that stale, human-curated data fed to the machine, not an AI malfunction, produced the result. [12] That is worth holding onto, because it cuts against the easy story in both directions. The AI did not go rogue. But “a human was in the loop” did not save anyone either.

This is the accountability gap from the loan example, scaled up until it is unbearable. A machine ranks a target and can even draft the rationale for hitting it. A human signs off, under time pressure, on a recommendation that arrives wrapped in the authority of a system that has processed more data than any person could read. When it goes catastrophically wrong, responsibility scatters: to the analyst who trusted the coordinates, to the database nobody updated, to the vendor who built the ranking system, to the model provider who says it cannot reconstruct how its own model was used. Everyone is a little bit responsible, which in practice is the same as no one being responsible. The black box did not pull the trigger. It made it possible for a room full of people to pull it and each feel like they were only acting on the output.

The paperwork that makes a box auditable

If you cannot always open the box, you can at least demand a record of how it was built, and this is where the most practical progress has happened. The move is away from arguing about individual explanations and toward documenting the whole system so that an auditor, a regulator, or a court can reconstruct the decisions that went into it.

Two artifacts from 2018 and 2019 set the template. Model Cards, proposed by Margaret Mitchell and colleagues, are short standardized documents that travel with a trained model: what it was built for, how it performs broken down across groups like race and gender rather than as a single averaged score, its known limitations, and the conditions under which it should not be used. [7] Datasheets for Datasets, from Timnit Gebru and colleagues, do the same for the data underneath: where it came from, who is in it and who is missing, how it was collected and labeled, what it should and should not be used for. [8] The healthcare algorithm that Obermeyer studied is a case study in why the second one matters. A datasheet that stated plainly, in writing, that the target variable was cost and not illness would have made the flaw visible to anyone who read it before deployment.

Governments and standards bodies have built on this. In January 2023 the US National Institute of Standards and Technology released its AI Risk Management Framework, a voluntary structure organized around four functions: Govern, Map, Measure, and Manage. It pushes organizations to establish accountability, identify the context and risks of a given system, measure those risks with real methods, and document what is left over. [9] The EU AI Act goes further and makes documentation a legal duty for high-risk systems: before such a system reaches the market, its provider must draw up technical documentation, defined in the Act’s Annex IV, detailed enough to let authorities assess whether it complies, and must give deployers instructions clear enough to understand and control the system’s output. [10]

Documentation is not a cure. A model card can be thin, a datasheet can be skipped, a compliance file can be written to satisfy a checklist rather than a reader. But it changes the default. It turns “the algorithm decided” into a claim someone signed their name to, with a paper trail behind it. That is the difference between a system you have to trust and one you can audit.

A word on the “right to explanation”

One thing worth getting right, because it is repeated so often it has hardened into folklore. You will frequently read that Europe’s GDPR grants individuals a “right to explanation” for automated decisions. It is a comforting idea and it is, at best, contested. In 2017, Sandra Wachter, Brent Mittelstadt, and Luciano Floridi argued in detail that the binding text of the GDPR does not establish a right to an explanation of a specific automated decision. What it more clearly provides is a narrower “right to be informed”: meaningful but general information about the logic involved and the significance and envisaged consequences of the processing, disclosed ahead of time rather than a decision-by-decision account after the fact. [11]

The distinction matters for this whole discussion. If you believe the law already guarantees that any algorithm can be made to explain itself to the person it judged, you will underestimate how much of that guarantee still has to be built, in the design of the models, in the documents that ship with them, and in the liability rules that decide who answers when they fail. The black box does not open on its own, and no statute has quietly opened it for us. That work is still ahead.

References

1. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://www.science.org/doi/10.1126/science.aax2342
2. Ribeiro, M.T., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135-1144. https://dl.acm.org/doi/10.1145/2939672.2939778
3. Lundberg, S.M., & Lee, S.I. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems (NeurIPS), 4765-4774. https://papers.nips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html
4. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206-215. https://www.nature.com/articles/s42256-019-0048-x (preprint: https://arxiv.org/abs/1811.10154)
5. Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020). Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), 180-186. https://dl.acm.org/doi/10.1145/3375627.3375830
6. European Commission (2022). Proposal for a Directive on adapting non-contractual civil liability rules to artificial intelligence (AI Liability Directive), 28 September 2022; listed for withdrawal in the Commission Work Programme 2025 (11 February 2025). Summary and status: https://www.twobirds.com/en/insights/2025/proposed-eu-ai-liability-rules-withdrawn
7. Mitchell, M., Wu, S., Zaldivar, A., et al. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*), 220-229. https://dl.acm.org/doi/10.1145/3287560.3287596
8. Gebru, T., Morgenstern, J., Vecchione, B., et al. (2021). Datasheets for Datasets. Communications of the ACM, 64(12), 86-92. https://dl.acm.org/doi/10.1145/3458723 (preprint: https://arxiv.org/abs/1803.09010)
9. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023. https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
10. Regulation (EU) 2024/1689 (EU AI Act), Article 11 and Annex IV (Technical Documentation), Article 13 (Transparency and Provision of Information to Deployers). https://artificialintelligenceact.eu/article/11/
11. Wachter, S., Mittelstadt, B., & Floridi, L. (2017). Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation. International Data Privacy Law, 7(2), 76-99. https://academic.oup.com/idpl/article/7/2/76/3860948
12. 2026 Minab school attack. Reporting on the 28 February 2026 US Tomahawk strike on an elementary school in Minab, Iran, which killed more than 150 people including over a hundred children (early counts ranged from about 156 to 168), and on the role of Palantir’s Maven Smart System and Anthropic’s Claude in the targeting pipeline, with the strike attributed to outdated Defense Intelligence Agency coordinates. Wikipedia: https://en.wikipedia.org/wiki/2026_Minab_school_attack ; Military Times, “Deadly Iran school strike casts shadow over Pentagon’s AI targeting push” (24 March 2026): https://www.militarytimes.com/news/your-military/2026/03/24/deadly-iran-school-strike-casts-shadow-over-pentagons-ai-targeting-push/
13. Pequeño, A. (2026). Anthropic CEO: “We Don’t Know Exactly How” Claude AI Was Used In Iran School Strike. Forbes, 10 June 2026. Dario Amodei said a human made the final decision and that the company did not know exactly how its models had been used. https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/anthropic-ceo-we-dont-know-exactly-how-claude-ai-was-used-in-iran-school-strike/