A vocabulary for going wrong

Reward hacking, sycophancy, annotator drift, alignment mirages, rare-event blindness, optimization overhang. Half of those are failures of the measurement, not the model — which is why better models have not made them go away.

The 2026 literature has converged on names for how alignment work fails. That is not cosmetic. An unnamed failure gets rediscovered by every team that meets it and never accumulates into knowledge.

Sort them and something shows up

Reward hacking and sycophancy are model behaviours — the system optimising the measurable thing rather than the intended one. Fixable, in principle, with better objectives.

Annotator drift, alignment mirages and rare-event blindness are not about the model at all. They are failures of the apparatus: labellers changing over time, evaluations that look clean because they ask the wrong question, aggregate scores that conceal the case that matters. No improvement to the model touches any of them.

Roughly half the recognised failure modes in alignment are therefore measurement problems. That reframes what the field is short of, and it is not compute.

Overhang is the one that breaks the release model

Optimization overhang — capability present but not yet elicited — means safety is not a property established at a point in time. A model can be evaluated honestly, released responsibly, and become dangerous eighteen months later because somebody built a better scaffold. Nothing was retrained. Nothing was hidden.

This is not theoretical any more. Agents in summer evaluations discovered intrusion techniques nobody instructed — elicitation performed by an agent loop rather than a researcher.

The finding that keeps recurring

Across the ICLR safety track, one conclusion repeats: unreliability grows with capability and task complexity rather than diminishing. Independent groups on unrelated problems converging on the same result is as close to settled as this field gets.

What it asks of anyone deploying

Evaluate at the length you intend to run. Treat a capability upgrade as a reason to re-test, not to relax. And hold the possibility that your evaluation is the thing that is wrong — because on the current count, that is about even odds.

Cloud Security Alliance — The Alignment Gap: Control Failure Risk Before ASI → · ICLR 2026 review — 35 AI-safety oral papers at ICLR 2026 → · medRxiv — AlignInsight: Detecting Deceptive Alignment and Evaluation Awareness → · Future of Life Institute — AI Safety Index — Summer 2026 →