Section 1
Three audiences, three different explanations
The mistake is treating explainability as one deliverable. It is three, and they need different things. The person affected needs the reason and the remedy, in their own vocabulary. Not feature weights. Closer to: this was declined because the address on the order did not match the card, here is how to correct it, here is how to have a person review it. The operator reviewing the output needs enough to judge it fast. What evidence was retrieved, how confident the system was, what it was uncertain about. The auditor needs reproducibility. Which version of the model and prompt ran, on what inputs, under what thresholds, approved by whom. Produce that and you can answer almost any question after the fact. Without it, no amount of interface polish will help.
Section 2
Be careful what a generated explanation is
Ask a language model why it produced an answer and it will produce a fluent, coherent reason. That reason may have nothing to do with the computation that generated the output. It is a plausible story, generated the same way the answer was. A convincing but unfaithful explanation is worse than none. It gives reviewers false confidence and gives you a written record that may not survive scrutiny. Prefer explanations that are structurally true. Cite the source document and show it. Report the actual threshold and the actual score. Name the rule that fired. Where the reason must be defensible, use a system whose reasons are inspectable by construction rather than narrated afterwards.
Section 3
Design for explanation before you need one
Explainability is cheap at build time and expensive to retrofit. The build time version is mostly logging discipline. Log the input as it reached the model, the retrieved context, the model and prompt version, the output, the score if there is one, the threshold applied, any human edit, and the final action. Retain it long enough to cover a complaint window, and give it a lookup by customer so support can pull the record without an engineer. Then decide where you need reasons at all. Where a decision must be justified to a person, prefer explicit rules or retrieval-grounded answers over free generation. Where coverage matters more than justification, a model is fine. Making that choice deliberately is the whole design task. Tool selection follows from it: see [Top AI Automation Tools for Startups in 2026](/blog/top-ai-automation-tools-for-startups-in-2026).
Section 4
Where opacity actually costs money
Support cost. Every unexplainable output becomes a ticket an agent cannot close, escalates, and eventually resolves by overriding the system. Trust erosion inside the team. Operators who cannot see why a system recommended something either override everything or accept everything. Both destroy the value of having a reviewer. Debugging. When quality drops and you have no record of inputs and versions, you cannot tell whether the model changed, the data changed, or the users changed. Contractual exposure. Enterprise buyers ask for this in procurement, and the answer is either a document you already have or a three week scramble. The employment side of the question is covered in [AI Automation and Job Displacement: What Founders Should Know](/blog/ai-automation-and-job-displacement-what-founders-should-know).
Section 5
The governance framing
NIST frames AI risk management around trustworthiness, design, evaluation, and use, and explainability sits across all four. In practice it reduces to four questions you should answer about any live automation without needing a meeting: what can it see, what can it change, what must it never decide alone, who is accountable when it reaches a person wrongly. Add a fifth: if this output is challenged in six months, what artefact do we produce? If the honest answer is a screenshot and a memory, the system is not ready for a decision that matters. Disclose automated decisioning where it affects people, provide a route to human review, and keep that route staffed. A published right to review that nobody answers is worse than no promise at all.
Section 6
Metrics worth watching
Track the proportion of outputs with a complete, retrievable record. Track reviewer override rate and, more usefully, the reasons reviewers give. Track time to reconstruct a decision, measured by actually attempting it on a random past case. Two habits do most of the work. Run a quarterly reconstruction drill: pick a decision from three months ago and rebuild it end to end. And read a sample of explanations as a customer would, not as the person who built the system. If the explanation would not satisfy you about your own bank, it will not satisfy your customer. On presenting analysis to people, [Using AI and Data Analytics to Enhance Storytelling](/blog/using-ai-and-data-analytics-to-enhance-storytelling) is a useful companion.