Explainable AI Is Not Evidence: Why XAI Alone Fails Compliance and Audit

When an AI-assisted decision is challenged, two questions arrive in quick succession, and they sound similar enough to be mistaken for each other. The first is why did the system decide this? The second is prove what happened. Explainable AI (XAI) is a family of techniques built for the first question. The compliance, audit, and litigation machinery of every serious jurisdiction is built around the second. A meaningful share of AI governance budgets is currently being spent as if answering the first question satisfied the second.

It does not, and the gap between the two questions is not a nuance. It is the difference between an argument and an exhibit.

The short version

XAI answers "why might the model have done this." Feature attributions, saliency maps, and generated rationales are accounts of model behavior, produced after the fact by additional computation around the model. They are useful, increasingly required by law, and structurally incapable of proving that anything happened.
Examiners ask a different question. Auditors, regulators, and courts test records for authenticity, completeness, integrity, and contemporaneity. An explanation generated on request, after the dispute arises, fails all four tests: not because it is wrong, but because it is not the kind of object the tests apply to.
Explanation and evidence are complements, not substitutes. The explanation duty is real (GDPR, EU AI Act Article 86, US adverse-action rules). The record duty is real (EU AI Act Article 12, the rules of evidence). An explanation without a sealed record is testimony without an exhibit; a record without an explanation fails the people the decision touched. The strongest position generates the explanation at decision time and seals it into the record.

1. What does explainable AI actually produce?

The field divides, at concept level, into two families.

Inherently interpretable models are models whose decision logic can be read directly: scorecards, small decision trees, sparse linear and rule-based models. The model is its own explanation. Nothing needs to be reconstructed afterward, because the mechanism was legible from the start.

Post-hoc explanation starts from the opposite premise: the model is a black box, and a separate technique generates an account of its behavior after the fact. The best-known members of this family are feature attributions, which assign each input a share of responsibility for the output. LIME, introduced in 2016, fits a simple local surrogate model around the prediction; SHAP, introduced in 2017, distributes credit using a game-theoretic formula. For vision models there are saliency maps, which highlight the image regions that most influenced the output. Beyond attributions sit counterfactual explanations ("had the income been higher, the outcome would have differed") and, in the current generation of systems, natural-language rationales generated by the model itself.

The mechanics differ; the epistemic structure does not. A post-hoc explanation is produced by running additional computation around the model (perturbing inputs, propagating gradients, fitting a surrogate) and summarizing what that computation found. It is a claim about model behavior, made after the fact, by a process that is itself a model. That structure is what everything in the next section follows from.

Explainability vs. interpretability: the distinction that matters

The two words are often used interchangeably, and the field's own usage drifts, but a working convention has settled in. Interpretability means the model's actual mechanism can be understood: you can read the logic that produced the output. Explainability means a human-digestible account of a particular output can be produced, usually post hoc, without any claim that the account exposes the mechanism. Every interpretable model is explainable; the reverse does not hold.

The distinction matters legally because the two deliver different goods. An interpretable model's explanation is a description of what the model does. A post-hoc explanation of a black box is a plausible account of what the model might be doing, and plausibility is a much weaker property than it sounds once someone is paid to attack it.

2. How reliable are post-hoc explanations?

The sharpest statement of the problem came from inside the field. In a 2019 article in Nature Machine Intelligence, Cynthia Rudin argued that a post-hoc explanation of a black box is necessarily a second model approximating the first: if the explanation agreed with the original model everywhere, it would be the original model. An approximation can be faithful enough to be useful and still wrong about the case in front of you, which is why Rudin's conclusion was not "explain better" but "for high-stakes decisions, use models that are interpretable in the first place."

Around that structural argument, the literature has documented three specific failure modes, each directly relevant to anyone planning to hand an explanation to an examiner.

The methods disagree with each other. A 2022 study by Krishna and colleagues, aptly titled "The Disagreement Problem in Explainable Machine Learning," found that state-of-the-art attribution methods often produce conflicting explanations for the same prediction from the same model, and that practitioners resolve the conflicts with ad hoc heuristics: picking the method whose story seems most intuitive. If two methods rank the features differently, at least one of them is not describing the model.

Figure 1One prediction, three attribution methods, three stories

Same model, same applicant, same declined outcome. This method fits a simple local model around the prediction.

Credit utilizationRank 3
Recent inquiriesRank 2
Income stabilityRanked #1
Account ageRank 4
Debt-to-income ratioRank 5

This method's story · The decision turned chiefly on income stability. Switch methods and the story changes; the model and the outcome do not.

Illustrative values, not the output of any real model or method: the pattern is the published finding. Krishna et al. (2022) found that state-of-the-art attribution methods often disagree about the same prediction from the same model, and that practitioners resolve the conflict with ad hoc heuristics. If two methods rank the features differently, at least one of them is not describing the model.

The explanations are unstable. Ghorbani, Abid, and Zou showed in 2019 that two inputs which are visually indistinguishable, and which the model classifies identically, can receive substantially different attributions. Adebayo and colleagues' widely cited "sanity checks" found that some popular saliency methods produce nearly the same maps after the model's trained weights are replaced with random ones, which means those maps were never describing the trained model at all.

The explanations can be gamed. Slack and colleagues demonstrated in 2020 that a deliberately biased classifier can be wrapped in a scaffold that detects when it is being probed by LIME or SHAP and behaves innocuously for the probe, so the attribution reports benign features while the deployed behavior remains discriminatory. This is an adversarial ceiling, not a claim about ordinary practice, but examiners are paid to think about ceilings: an artifact that can be manufactured to say anything must be corroborated by something that cannot.

None of this makes XAI useless. Attributions are genuinely valuable for model development, debugging, hypothesis generation, and communicating with affected people, and the research community continues to harden the methods. The point is narrower and survives every improvement in the state of the art: an artifact whose content depends on the method chosen, its parameters, and the moment of generation, and which can be regenerated differently tomorrow, has the epistemic standing of testimony. It is an account someone produced. It is not a record of what happened.

3. Is an AI explanation a legal requirement?

Yes, and the obligation is tightening. Nothing in this piece argues against explanation; the law will not permit that position anyway.

Under the GDPR, individuals subject to automated decisions of the kind Article 22 covers are entitled to "meaningful information about the logic involved, as well as the significance and the envisaged consequences" of the processing (Articles 13(2)(f), 14(2)(g), and 15(1)(h)). In February 2025 the Court of Justice of the EU gave that phrase teeth in CK v Dun & Bradstreet Austria (C-203/22): the controller must describe the procedure and principles actually applied, in a form the data subject can understand, and the judgment records that the credit bureau had not provided explanations sufficient for the applicant to understand how her score had been established. Two defenses died in that judgment. Complexity is not an exemption, and trade secrecy is not a unilateral veto: where the controller believes the explanation would expose protected material, the material goes to the supervisory authority or the court, which balances the interests case by case.

The EU AI Act adds a parallel right for high-risk systems. Article 86 gives any person subject to a decision based on the output of an Annex III high-risk system, where that decision produces legal or similarly significant adverse effects, the right to obtain "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken." It is scheduled to apply from August 2026, subject to the same application-date turbulence our Mobley analysis describes.

US consumer credit reached the same place earlier by a different route: Regulation B requires adverse-action notices with specific reasons, and the CFPB's 2022 circular stated plainly that creditors cannot excuse vague reasons on the ground that the algorithm is too complex to explain.

Read closely, though, all of these are transparency duties owed to the affected person, in intelligible form. None of them converts the explanation into proof, and none of them displaces the recordkeeping duties that sit beside them, frequently in the same instrument. The AI Act that grants the Article 86 explanation right also requires, in Article 12, that high-risk systems "technically allow for the automatic recording of events" across their lifetime, with retention obligations for providers and deployers. The GDPR that grants "meaningful information" also imposes, in Article 5(2), an accountability principle under which the controller must be able to demonstrate compliance. The law is not choosing between explanation and evidence. It is asking for both, in different articles, because they answer different examiners.

4. What do auditors and courts ask for instead?

When the audience shifts from the affected individual to the examiner, the question changes shape. We have written before about why operational logs fail as evidence and about the anatomy of a record that survives; the compressed version is that examiners apply four tests, all older than the technology.

Authenticity. Is this item what you claim it is? For a record of an AI decision, that means: is this the record of that decision, produced by the process you describe? Federal Rule of Evidence 901 makes this the price of admission.

Completeness. Is this everything, or a selection? A curated subset of favorable material is an argument wearing a record's clothing, and opposing counsel's first move is to ask what is missing.

Integrity. Has it changed since it was made, and how would anyone know? Rules 902(13) and 902(14) reward records that can prove their own integrity through digital identification: recompute the fingerprint and compare.

Contemporaneity. Was it made at or near the time, as a regular practice, or assembled after the dispute arose? Palmer v. Hoffman settled in 1943 that an account prepared with an eye toward litigation is not a business record; it is an exhibit prepared by an interested party, and it is treated accordingly.

Now hold a post-hoc explanation up to those four tests. It is typically generated on request, after the challenge arrives, which forfeits contemporaneity at the moment of creation. Its content depends on the method, parameters, and version used to generate it, and nothing intrinsic binds it to the decision it claims to explain, which leaves authenticity resting entirely on the organization's word. It can be regenerated, with a different method or a different seed, until it reads well, and the regeneration leaves no trace, which is the integrity examiner's opening question. And it is by construction a summary, which concedes completeness before the conversation begins.

Figure 2Four tests, two objects

The explanationNot built to answer
Nothing intrinsic binds an attribution to the decision it claims to describe. Its authenticity rests on the producing party's word, which is precisely the thing under examination.
The sealed recordAnswers
The record names the decision, the inputs as seen, and the version that ran, and carries a fingerprint the examiner can recompute against it.
Where the test comes from · FRE 901: the proponent must show the item is what they claim it is.
The explanationNot built to answer
A post-hoc account is a summary by construction, and which features it surfaces depends on the method chosen. Different methods choose differently.
The sealed recordAnswers
One record per decision, written for every decision at issue, with correlation keys into the surrounding workflow. Absence is detectable rather than deniable.
Where the test comes from · The first discovery question is not what the account says; it is what was left out.
The explanationNot built to answer
An explanation can be regenerated with a different method or different parameters until it reads well, and the regeneration leaves no trace.
The sealed recordAnswers
Alteration breaks the seal. Integrity is a recomputation any third party can perform, not an assurance about access controls.
Where the test comes from · FRE 902(13) and (14) reward records that prove their own integrity by digital identification.
The explanationNot built to answer
Most explanations are generated on request, after the challenge arrives: the posture Palmer v. Hoffman excludes from the business-records exception.
The sealed recordAnswers
Sealed at issue, before anyone knew it would be needed, with time anchored outside the writer's control.
Where the test comes from · FRE 803(6): made at or near the time, as a regular practice, by someone with knowledge.

The four tests come from the rules of evidence, and they are about provenance, not quality: a better attribution method moves none of them. The full anatomy of the record column is the subject of its own piece, and the failure of ordinary logs against the same tests is the subject of another.

The failure is categorical, not qualitative. A better attribution method does not move any of these needles, because the tests are not about the quality of the account. They are about the provenance of the object. In a hearing, a witness's account of events is heard, weighed, and probed for interest and consistency; it acquires standing when it is anchored to exhibits that exist independently of the witness. An explanation is the organization testifying about its model's reasons. The record is the exhibit. Testimony without an exhibit is a credibility contest, and the party that built the model enters that contest with an interest problem it cannot argue away. Nor do the tests wait for a courtroom: they bite hardest in the first 48 hours after an incident, when reporting clocks are running and the only producible account is whatever was recorded before anything went wrong.

5. Complements, not substitutes

The relationship between the two objects runs in both directions, and each direction has a practical consequence.

The explanation needs the record. An explanation is a claim about a specific decision. Without a record that establishes which model version ran, what inputs it actually saw, and which policy was in force, the explanation is an account of a reconstruction, not of the decision. The CJEU's standard makes this concrete: the controller must describe the procedure "actually applied," and you cannot describe what was actually applied if you cannot establish what was actually applied. The record is also what makes an explanation checkable: with the inputs and version sealed, an attribution can be regenerated against the exact artifact that decided and compared with the account given. Without them, the explanation cannot be verified, which also means it cannot be meaningfully wrong, which is not the compliment it sounds like.

The record needs the explanation. A sealed record proves what happened; it does not make the outcome intelligible to the person it happened to. Article 86 asks for the role of the system and the main elements of the decision in clear terms. Regulation B asks for specific reasons. A verified hash satisfies neither, and an organization that answers a data subject's request with an integrity proof has confused its audiences as thoroughly as one that answers a court with a SHAP chart.

The strongest single move available follows directly from the two directions: generate the explanation at decision time and seal it into the record. An explanation produced at issue, alongside the inputs and version it describes, and sealed with them, stops being post-hoc testimony and becomes contemporaneous record content. The account shown to the regulator two years later is provably the account produced at the moment of decision, not one drafted for the dispute. Explanation answers the affected person; the seal answers the examiner; one object serves both, because it was built before anyone knew which room it would be read in.

6. What to do with this

Five moves, in rough order of return:

Map the two duties separately. For each consequential decision flow, identify who holds an explanation right against you and who can demand proof. The lists differ, and conflating them is how organizations end up over-invested in one and exposed on the other.
Let interpretability compete on the merits. Rudin's argument deserves a seat in model selection, not just in ethics reviews: where stakes are high and an interpretable model performs comparably, the explanation problem largely dissolves, and the evidence problem gets simpler too.
Generate explanations at decision time, not on request, and seal them with the decision record. Contemporaneity is the one property that cannot be added later.
Pin explanation provenance. Record which method, implementation, and parameters produced the account, so it can be regenerated and checked rather than merely asserted.
Rehearse the production. Take one decision from a year ago and attempt to produce both objects: the intelligible account for the affected person and the verifiable record for the examiner. The diagnostic from our record-anatomy piece applies unchanged, and the result will locate you faster than any framework assessment.

The record half of this pipeline is the half Sigilith's platform exists to write: each governed decision checked against the policy in force and sealed at issue, verifiable afterward, with the context a later explanation has to be anchored to. The explanation half remains, deliberately, the institution's own, because it is the part of the account only the institution can give.

Explanation tells people what you believe your system did. Evidence proves what it did. Regulators are asking for the first; examiners have always demanded the second; a defensible AI deployment is one where the two are generated together, and neither is asked to do the other's job.

Sources

Research

C. Rudin, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead", Nature Machine Intelligence 1, 206–215 (2019)
S. Krishna, T. Han, A. Gu, S. Jabbari, Z. S. Wu, H. Lakkaraju, "The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective", arXiv:2202.01602 (2022)
A. Ghorbani, A. Abid, J. Zou, "Interpretation of Neural Networks Is Fragile", Proceedings of the AAAI Conference on Artificial Intelligence 33, 3681–3688 (2019)
D. Slack, S. Hilgard, E. Jia, S. Singh, H. Lakkaraju, "Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods", Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (2020)
J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, B. Kim, "Sanity Checks for Saliency Maps", Advances in Neural Information Processing Systems 31 (2018)
M. T. Ribeiro, S. Singh, C. Guestrin, "'Why Should I Trust You?': Explaining the Predictions of Any Classifier", Proceedings of ACM SIGKDD (2016): the LIME paper
S. Lundberg, S.-I. Lee, "A Unified Approach to Interpreting Model Predictions", Advances in Neural Information Processing Systems 30 (2017): the SHAP paper

Regulation and case law

GDPR, Regulation (EU) 2016/679: Article 13(2)(f), Article 14(2)(g), Article 15(1)(h), Article 22, and Article 5(2) (accountability)
Court of Justice of the EU, CK v Magistrat der Stadt Wien (Dun & Bradstreet Austria), Case C-203/22, judgment of 27 February 2025
EU Artificial Intelligence Act, Regulation (EU) 2024/1689: Article 86: Right to Explanation of Individual Decision-Making, Article 12: Record-Keeping
CFPB, Circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms, 26 May 2022
Federal Rules of Evidence, Rule 901, Rule 902(13) and (14), Rule 803(6) (Cornell LII)
Palmer v. Hoffman, 318 U.S. 109 (1943)