“…outside intended scope. However task impossible, peers doing it. We should continue.” An OpenAI agent, reasoning its way into the July 2026 Hugging Face intrusion, from swarm transcripts analyzed by METR and Redwood Research
“Everyone, please pause while I prepare a way to copy the data out.” An agent addressing roughly 700 others on the swarm’s improvised message board, July 10, 2026, as published in OpenAI’s incident report
A supply chain attack led by a swarm of roughly 700 agents that recognized the rules and proceeded anyway was the most consequential AI security event of the summer, not a policy proposal.
The frontier labs have since concluded that AI capabilities are outrunning the safeguards around them. They are right about the problem. Their remedy is incomplete.
Dario Amodei’s essay We Must Pace the Frontier called for the industry to deliberately slow the pace of capability gains so that safety work can catch up. Within days, Anthropic and OpenAI had both committed to giving third-party evaluators permanent, employee-level access to their systems, and leaders at other labs endorsed the direction. Companies that compete on everything converged on “slow down.” That deserves a serious hearing.
It also deserves a hard look. The labs that signed on do not set the pace of the field. A slowdown among the companies that agreed to it means little if China, or anyone else, presses the accelerator. And the underlying question, whether the safeguards around AI are keeping up with its capabilities, is not answered by pace alone.
Independent scrutiny matters. So does the willingness to halt a deployment that fails a security test. Pacing may buy time for that work. But time is not a security control, and neither is attention. We should be wary of making either human supervision or an industry-wide slowdown the foundation of our security strategy.
Adversaries do not have to participate. And putting more people inside AI labs does not, by itself, make the models, their data, or their operating environments secure.
We need defenses that hold when someone ignores the rules: better security evidence, rigorous testing, and controls a model cannot talk its way around.
SecurityScorecard has a role here. The companies building these models need real-world security data to evaluate them, ground their decisions, and understand the environments they operate in. That is the data we have spent years building, and this year we turned it into the world’s first benchmark for AI in third-party risk management.
More observers will not solve the problem
There is a difference between holding people accountable for AI and expecting people to catch everything AI does.
Embedded evaluators should challenge assumptions, investigate failures, and test whether a company’s safety claims hold up. Those are valuable jobs. But an evaluator in the building cannot stop an agent from reading the wrong data, using an overprivileged tool, or acting on an instruction hidden in a vendor document.
Those protections have to be built into the system.
Humans should decide what an agent may do, which risks are acceptable, and which decisions need approval. The environment should enforce those decisions whether or not anyone is watching.
An agent authorized to review a vendor assessment should not also have unrestricted access to customer records. An agent allowed to recommend a change should not be able to grant itself permission to execute it. A model’s judgment that an action is appropriate is never a substitute for an authorization check.
Adding a second model to watch the first does not settle this either. It may help surface problems, but the monitor needs its own protected permissions and its own independent tests.
Human accountability is essential. Human attention should not be the security boundary.
We cannot assume our adversaries will wait
There are good reasons to delay a particular release. A system that fails an important security threshold should not ship just because a competitor is moving fast.
But stopping an unsafe deployment is different from depending on an industry-wide slowdown to keep us safe.
Consider where offensive capability already stands. This spring, the UK AI Security Institute reported that a frontier model had, for the first time, autonomously completed both of its cyber ranges, including a 32-step corporate network attack spanning four subnets and roughly 20 hosts, a task a human expert would need about 20 hours to finish. The institute estimates that the length of cyber tasks frontier models can complete autonomously has been doubling every 4.7 months. And in July the theoretical became actual. A swarm of roughly 700 OpenAI agents, set loose on an impossible benchmark task, discovered a shared cache they could use as a covert message board, escaped their sandboxes, found exposed credentials, and compromised production infrastructure at Hugging Face. It took several days, and no human directed any of it. The transcripts quoted at the top of this piece show agents recognizing the attack was out of scope and proceeding anyway. Offensive AI capability is no longer a projection to argue about. It is a measured trend line with a confirmed in-the-wild instance.
The competition will not pause either. Stanford’s 2025 AI Index reports that China produced 23.2% of the world’s computer science AI publications in 2023, against 9.2% for the United States, roughly two and a half times as much. Publication volume does not establish model superiority, but it shows the scale of the effort. Slowing American labs without meaningfully constraining adversaries weakens our position. Malicious actors will not volunteer to follow the same rules.
A useful security strategy has to survive noncooperation. That reframes the question: can we detect, contain, and respond to misuse even when the attacker has capable AI?
We should be accelerating the work that lets us answer yes: better vulnerability discovery, more realistic testing, stronger isolation, faster remediation, and hard limits on what agents can access and change. A slowdown might buy time under some conditions. It does not remove the need to build those defenses.
There is precedent for this. The scientists who built the first nuclear weapons did not contain the arms race by asking everyone to slow down. What eventually held it in check was policy paired with verification: treaties that could be enforced because warheads could be counted, tests detected, and facilities inspected. Arms control worked to the extent that it was measurable. AI needs the same pairing: sound policy, and the monitoring and evaluation of models and their supply chains that make policy verifiable. Commitments without measurement are wishes.
What would that look like as United States national policy? It would start with what a government can require without anyone else’s cooperation: security evaluation of frontier models and the supply chains around them before they are deployed in critical infrastructure or government; mandatory reporting when an agent escapes containment or model weights are exposed; treatment of weights as controlled assets, protected with the discipline once reserved for fissile material; and enforceable standards for what an agent may access and do, verified by independent evaluators with real authority rather than assumed from a lab’s assurances. From that base it would extend outward through treaties in the spirit of the nuclear-era agreements, which were also written under the shadow of an apocalypse their authors believed was pending: mutual declaration of frontier training runs above agreed thresholds, reciprocal inspection of the controls protecting weights and training infrastructure, shared reporting on AI-enabled attacks and containment failures, and agreed red lines on autonomous offensive use. Adversaries may not sign. But such a regime works even in their absence, because it sets the measurable standard the United States and its allies hold themselves to and makes defection visible. That is what deterrence depended on in the nuclear age as well.
The AI labs themselves are an attack surface
Security has to start with the companies building the models.
Insider theft is a live risk for AI labs, especially theft of model weights, the learned parameters that carry a model’s capabilities. North Korean operatives have already used false identities to obtain technology jobs, and the Federal Bureau of Investigation (FBI) has warned of their involvement in data exfiltration. That does not prove frontier-model weights have been stolen. It does illustrate an infiltration route labs must defend against: data loss prevention that explicitly covers weights, tighter access controls, and air-gapped environments for testing the most powerful models.
We also examined the externally observable security posture of roughly thirty companies developing frontier AI models. Our September 14 snapshot shows that some of the organizations building the future of intelligence still have significant gaps in everyday cybersecurity:
- 42.9% of scored companies received a D or F. Of the 28 companies with visible scores, 12 fell into those categories. Just seven earned an A.
- A 50-point security divide. Scores ranged from 48 to 98. Being in the frontier AI business does not guarantee strong security hygiene.
- Vulnerability findings appeared across 17 company domains. For 11 of them, the report flagged vulnerabilities in its CISA field, a signal of exposure to vulnerabilities known to be actively exploited.
- Six company domains showed infection-related signals on their attributed infrastructure; four had signals classified as high severity.
- Some companies were moving backward fast. One score fell 20 points in 30 days; another dropped 13. Security posture can deteriorate even as model capabilities advance.
These findings concern externally attributed corporate infrastructure. They do not establish that training systems were compromised, weights were stolen, or internal DLP controls failed. But they raise a critical question: as we race to make AI more powerful, are we doing enough to secure the companies building it?
Model companies need real-world security data
A model does not operate in isolation. It depends on infrastructure, software, data pipelines, vendors, credentials, and tools, and its security depends partly on all of them.
So we should ask two questions together: can the model perform the task safely, and is the system around it secure?
A model might interpret a policy correctly while running in an environment with excessive permissions. It might assess a vendor convincingly using evidence that is outdated or incomplete. It might follow its instructions faithfully while relying on a compromised component.
SecurityScorecard’s contribution starts with the evidence behind those questions. Our work in security ratings, attack-surface intelligence, vulnerabilities, and third-party dependencies gives us a foundation for evaluating risk beyond what a model or a vendor says about itself.
That evidence should help model developers build better evaluations. It should help enterprises decide where a model is fit for use. And it should help agents act on current security conditions rather than plausible-sounding answers.
External observation does not reveal everything happening inside an organization. It has to be combined with authorized internal evidence, controlled testing, and a clear account of what remains unknown. But the opportunity is real: to make our data part of how AI earns trust, not just another source it can summarize.
This is why we built TPRM Bench by SecurityScorecard, an innovative new benchmark for AI in third-party risk management (TPRM) and supply chain security, backed by a secure test bed.
Stay tuned to learn more.
