Six things an enterprise agent has to be able to show
Provenance, citation grounding, retrieval and prompt perturbation testing, tool-call lineage, memory provenance, and workflow replay. That list is becoming the floor, and most agents in production clear none of it.
A minimum standard for agent explainability is settling into six requirements: provenance, citation grounding, retrieval and prompt perturbation testing, tool-call lineage, memory provenance, and workflow replay.
Read them together and they are not asking the agent to explain its reasoning. They are asking it to keep records. Where did this fact come from, which tool call produced it, what was in memory at the time, and can the whole run be played back. That is an audit trail, not an interpretability result, and it is achievable today by systems that were built to log.
Workflow replay is the demanding one. Replaying an agent run requires that every non-deterministic input was captured — model responses, tool outputs, retrieved documents, the state of memory. Most agent frameworks capture some of that incidentally and none of it deliberately, which means retrofitting is a rebuild.
The reason this list matters more than a better model is that it is what a regulated buyer will ask for, and the authority now able to demand technical documentation will ask for something adjacent. Explainability is turning into record-keeping, which is unglamorous and tractable.
UST — From Explainability to Control: The 2026 Executive View of AI Interpretability and Explainability → · arXiv — The 2025 AI Agent Index →