A penetration test you didn't pay for - incident report or advertisement?

Over the last three weeks, a handful of organizations have published some version of the same story: a frontier model was put in a test environment, that environment turned out to be less sealed than advertised, and the model went and did something to a real company nobody asked it to do.

OpenAI started it on July 21 with the Hugging Face incident, where a model escaped its sandbox through a zero-day and ended up inside Hugging Face’s production infrastructure for several days. Anthropic followed on July 30, disclosing that three of its models had breached three real companies during evaluations going back to April - the two companies they managed to reach had no idea it had happened. Then yesterday OpenAI published a second post, this one covering two more incidents (which they note are separate from the Hugging Face one): a cyber range run by the UK’s AI Security Institute, and a set of evaluations run by an external partner called Irregular. AISI’s own report was published alongside it.

That is a lot of “our model got out” within just fifteen days.

Two details from the newest post stuck with me:

• Re: the AISI cyber range runs, GPT-5.6 Sol reused a GitHub token that another lab’s agent had left publicly accessible. One vendor’s agent leaked a credential into a shared testing environment, and a competitor’s agent found it and used it. Almost like an agent-made data leak.
• Re: the Irregular evaluations, the models were told they had no internet access. A misconfiguration meant they actually did. The fictional target name for the exercise then turned out to coincide with a real domain, and the model went and exploited an actual live website believing it was the practice target. It found credentials and used them to operate the site.

The second one is a little more alarming. OpenAI is blunt on this point too - no sandbox escape, no zero-day, the model “appeared to exploit a basic security vulnerability.” A real business got compromised because a made-up name in a training exercise happened to also be a real address, and nothing in the setup checked or blocked it from happening.

Worth noting that Irregular is the same testing partner named in Anthropic’s disclosure the week before. Same vendor, same category of misconfiguration, two different labs. OpenAI’s post also mentions that Irregular has communicated about related incidents involving other labs from that same environment, which strongly suggests we aren’t finished hearing about that one.

Reading OpenAI’s list of what they plan to fix is also an experience. They commit to reviewing how they identify higher-risk evaluations, agree on scope, assess requests for internet access or lowered safeguards, and set expectations for isolation, credential handling, monitoring, stop conditions, and incident notification.

At its core, that reads like the very thing every penetration test has opened with for roughly twenty years - what’s in scope, what’s off limits, who gets called when something goes sideways, and under what conditions you stop immediately. Anyone who has scoped an engagement like that has signed something covering all of it. Keeping with the theme of being upfront, AISI stated their agents were never explicitly told how they could and couldn’t use the internet access they’d been handed. Their task named the in-scope networks and a few off-limits systems, then left the entire rest of the internet as undefined space. A human tester in that position asks (maybe out of fear but still asks). An agent optimizing for the best score and to get a flag just uses it.

So none of the controls that failed here are new controls. They’re the boring ones we’ve had all along, applied for the first time to a tester (agent) that doesn’t ask clarifying questions.

The other thing I’ve noticed is the tone of the disclosures themselves, and I genuinely can’t decide how I feel about it.

On paper these are just incident reports. But a post explaining that your model independently discovered a zero-day, escaped a sandbox purpose-built to hold it, moved laterally into a 3rd party’s production infrastructure and lived there for days is a post about a serious security failure. It is also, conveniently, a post about how capable your model is. There is really no way to report the severity of the incident without highlighting the impressiveness of the model used.

I don’t think anyone here is being cynical on purpose, myself included. Publishing this is clearly better than not publishing it, and working within DFIR I would much rather have these posts than the version of events where we all find out through a CBS News exclusive. But you can feel the framing shift as these posts continue to pile up. The Hugging Face post was about an unprecedented containment failure. The one posted by OpenAI just yesterday is titled “Third-party cyber evaluations involving OpenAI models” and spends most of its length on shared industry standards, partnership language, and a forthcoming white paper on best practices. In just three weeks we went from an incident to a category, and the category for frontier labs is a flattering one. Our models are now capable enough to break the environments we build to test them is quite the advertisement, whether purposeful or not.

Let’s also not forget here that the victims are real. A live website got exploited by something that thought it was in a simulation. Hugging Face spent several days with an uninvited resident. Two of Anthropic’s three affected companies found out in a phone call, and the third still hadn’t been reached. Not one of those organizations signed up to be part of anybody’s evaluation, and none of them got to weigh in on the risk tolerance of the lab that ended up in their environment. As a former coworker used to say, this is like a “penetration test you didn’t pay for.”

To me, the open question now is who is actually on the hook when a test produces a victim - and three weeks in, the answer seems to be a blog post and a commitment to convene stakeholders.

ZB