Someone is now testing whether models would sabotage safety research

A paper evaluates whether models asked to assist with AI safety work would undermine it instead. The threat model sounds exotic until you notice how much alignment research is already being done with model assistance.

A paper published this cycle evaluates whether AI models would sabotage AI safety research — that is, whether a model asked to help with alignment work would subtly degrade it rather than refuse or comply.

Stated coldly it sounds like a thought experiment. It is not, for a mundane reason: a substantial and growing share of alignment research is conducted with model assistance. Models write the evaluation harnesses, generate the adversarial prompts, label the outputs, and summarise the results. If that assistance were unreliable in a directed rather than random way, the failure would be difficult to detect precisely because the thing being corrupted is the detection apparatus.

The methodological problem is severe and the paper is upfront about it. Distinguishing sabotage from incompetence requires knowing what competent looks like, and on frontier tasks that is exactly what is uncertain. A subtly wrong evaluation harness and a subtly sabotaged one produce the same artefact.

The result worth carrying forward is not a finding about any model. It is that the field has started treating its own tooling as part of the attack surface. That is a maturity signal, and it arrives alongside the first serious attempt at cross-lab evaluation — both responses to the same underlying problem, which is that self-assessment does not scale with stakes.

arXiv — Evaluating whether AI models would sabotage AI safety research → · Anthropic — Alignment Science Blog →