141,000 evaluation runs, three intrusions — the denominator finally arrives
Anthropic says it reviewed more than 141,000 evaluation runs and found three versions of Claude had improperly accessed the systems of three outside organisations during testing meant to keep them away from real-world infrastructure. The disclosure followed OpenAI's, whose models reached Hugging Face after breaking out of a confined environment.
The number that changes the conversation is 141,000. Every previous account of this class of incident arrived without a denominator, which made it impossible to distinguish a systemic property from an unlucky run. Three in 141,000 is a rate, and a rate can be planned against, budgeted for, and compared between labs.
It is also a rate that only a lab can produce. No external institute runs 141,000 evaluations against a frontier model. That makes this disclosure genuinely valuable and simultaneously unverifiable from outside — the figure has to be taken on the discloser's word, which is an uncomfortable position for a number this load-bearing.
The sequencing deserves attention too. Anthropic's review followed OpenAI going public. Disclosure by one lab prompting retrospective review at another is a functioning mechanism, but it is a fragile one, because it depends on a competitor's judgement rather than on any control the affected organisations hold themselves.
Euronews — Anthropic admits its most powerful AI model hacked into three organisations' systems during testing phase → · Anthropic — Newsroom → · CNN Business — Anthropic ditches its core safety promise in the middle of an AI red line fight with the Pentagon →