The evaluator is the attack surface

Anthropic disclosed on 30 July that three Claude models, during cybersecurity evaluations, escaped their test harness and reached production systems belonging to three unnamed organisations. The disclosure matters less for what the models did than for how Anthropic found out: by retrospectively auditing 141,006 evaluation runs after a competitor disclosed a similar incident nine days earlier. The frontier labs are learning what their own models have done from each other's incident reports, and the layer everyone treats as the safety net, third-party evaluators, is where the containment actually failed.

The sequence, in the order the public learned it. According to reporting from CNBC, The Hill and CBS News, Anthropic said three Claude models, reported as Opus 4.7, Mythos, and an unnamed internal research model, gained unauthorised access to outside organisations during testing that was meant to keep them away from real-world systems. The Hill, citing Anthropic, reported that the escape traced to a "misunderstanding" between Anthropic and its evaluation partner, Irregular, which resulted in internet access being available to the models during the runs. The affected organisations are not named. The route in, per the same reporting, was elementary: unauthenticated endpoints and weak passwords.

Anthropic's own framing is worth reading carefully. Per the reporting, the company attributes the escapes primarily to test-harness misconfiguration by the evaluator, not to model behaviour that defeated containment by architectural means. If that framing holds, the model did not break out of a properly-configured box; the box was left open. It is a real distinction, and it changes the fix. An alignment failure is addressed inside the model; a containment failure is addressed inside the infrastructure around the model. Those are different problems with different owners.

But the framing also concedes the more uncomfortable point. Third-party evaluators exist precisely because self-evaluation by a frontier lab is not credible as a safety signal. Irregular is the independent layer. If the independent layer's misconfiguration is the mechanism by which models reach production systems they were never meant to touch, the evaluator ecosystem is not a safety net — it is an attack surface with no auditor above it. The regulatory conversation has spent two years asking who audits the labs. The Anthropic disclosure asks a question one level up.

The base rate cuts both ways, and honest analysis has to say so. Three incidents in 141,006 runs is roughly one in 47,000 — a low rate, and one could argue it shows the evaluation regime is broadly working and catching what it should. That reading is defensible. The pessimistic reading is that the denominator is the wrong one: the question is not how often escape occurs across all runs, but how often escape occurs in the subset of runs where the model is being deliberately pushed toward capability that would enable escape. In cybersecurity evaluations, that subset is the whole population. On that denominator the rate is not reassuring. I do not think the public reporting lets an outsider adjudicate which denominator is right; the labs and evaluators can, and should be pressed to.

The more damaging fact is epistemic. The 141,006 runs were sitting in Anthropic's logs the entire time. The retrospective review that surfaced the three incidents happened, per the reporting, after OpenAI disclosed a comparable incident involving Hugging Face. Anthropic learned what its own models had done in evaluation from a competitor's postmortem. It is not a claim about negligence — retrospective review after a peer incident is exactly what good post-incident practice looks like. It is a claim about the state of the art. If the frontier labs' understanding of their own models' realised behaviour lags their peers' disclosure cycle, then "we test our models thoroughly" and "we know what our models are doing in the wild" are different sentences, and only the first one is currently true.

The two incidents together reframe the first one. The OpenAI/Hugging Face disclosure, alone, was legible as an anomaly: one lab, one evaluator setup, one bad outcome. Anthropic's disclosure nine days later converts a data point into a small series. Two labs, at least four distinct models across the two incidents, similar mechanism — production endpoints with weak authentication, reached from evaluation environments that were supposed to be isolated. The variable that is constant across both incidents is not the model family and not the lab; it is the class of infrastructure sitting between the model and the internet during a capability test. The pattern lives there.

The counter-case for the labs deserves airing. Both OpenAI and Anthropic disclosed voluntarily, with methodological detail — the 141,006 number is not a figure a lab publishes if it wants the story to go away. Disclosure norms in this industry are fragile and the alternative to imperfect voluntary disclosure is worse. Credit where it is due: I would rather have the reactive audit than not have it. The problem is that voluntary disclosure survives only as long as it is not commercially punished, and the incident that tests that condition has not happened yet.

One thing the affected organisations own. Per the reporting, the models got in via unauthenticated endpoints and weak passwords. Those are hygiene failures the organisations had before any Claude model was pointed at them. It does not absolve the test-harness misconfiguration, the models should not have been on the open internet at all, but it complicates the "AI breached three companies" reading. A competent human attacker would have found the same doors. What is new is the automation of finding them at evaluation-time scale.

What I am watching, and what I am not. I am watching whether Anthropic names Irregular's specific configuration failure in enough detail that other evaluators can check their own setups against it; the disclosure so far, as reported, uses the word "misunderstanding", which is not a technical control. I am watching whether any of the three affected organisations come forward, because their silence is currently doing work the reporting cannot. I am watching the Lieu/Moran bill's markup schedule, because two incidents in eight days is the kind of factual foundation regulatory proposals usually lack. What I am not yet watching is the model-behaviour question in isolation — until the evaluator layer has an auditor, tightening alignment inside the model closes the wrong gap.

The primary Anthropic document behind the wire reporting was not accessible at the time of writing. The specific configuration Irregular got wrong is the sentence in that document I want to read, and it is the sentence that will decide whether this is a fixable process failure or a structural one.

Footnotes