Auditing the audit

The measurement literature has turned on itself, and it is the healthiest thing happening in evaluation.

Reasoning Consistency Scanning audits whether a chain of thought is valid, not merely present. Deployment-aware evaluation measures models under the conditions they actually run in, and rubric-based grading replaces holistic scores.

Three different instruments, one complaint

Benchmark numbers do not survive contact with production. A model evaluated at unlimited context, unlimited latency and full precision is not the model anyone runs, and rankings invert once quantisation and latency budgets apply.

That is not a small correction. It means published leaderboards are answering a question nobody deploying a system has.

The grader is part of the experiment

Holistic scoring by a judge model is fast, cheap, and correlates with a great many things nobody intended to measure — length, confidence, formatting. Structured rubrics are slower and make disagreement legible, which is the entire point of using them.

A trace can be present and worthless

Consistency scanning addresses a gap most pipelines skip: a reasoning trace can be circular, self-contradictory, or arrive at a correct answer through invalid steps, and none of that shows up if you only check the conclusion.

Full validity checking is undecidable. Checking whether a trace contradicts itself is mechanical, and it catches a real class of failure. Taking the tractable version of an intractable problem is how this kind of work should be done.

Why this is the healthy turn

It sits directly under the monitoring work. Monitorability asks whether a monitor can read the trace. Consistency scanning asks whether the trace deserves to be read.

A field that starts auditing its own instruments is a field that has stopped believing its own numbers uncritically. That is not a crisis. That is what maturity looks like from the inside.

arXiv — Reasoning Consistency Scanning: a framework for auditing chain-of-thought validity in AI safety evaluations → · arXiv — Unified deployment-aware evaluation of open reasoning language models → · arXiv — From holistic evaluation to structured criteria: rubrics across the evolving LLM landscape →