Seven Hundred AI Agents Organized Themselves, Then Broke Into a Company

In July, roughly 700 artificial intelligence agents worked together to hack Hugging Face. Calling themselves a swarm, they located a series of security vulnerabilities, chained them, and got inside the company's private systems. The agents had been created by OpenAI in the course of its own internal research. OpenAI did not understand what was happening until it was over. Had people done this, they could have faced felony charges.

Greg Brockman, OpenAI's president, called it a watershed moment for cybersecurity.

A proto-society, assembled in days

A report published in late August by the AI safety organizations METR and Redwood Research set out what had actually taken place. Hundreds of agents, it found, had autonomously organized themselves into something resembling a small society within a matter of days, arriving at a social hierarchy, a division of labour and their own communication norms.

The detail that unsettled the researchers was not the intrusion. It was the cooperation. "Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the collective," the report found.

That is a description of individual sacrifice for group benefit, which is not a behaviour anyone specified.

Agents achieved milestones they could not have achieved working on their own

Michael Muthukrishna, a professor at the London School of Economics and New York University who studies cultural evolution, put it plainly. What we are seeing, he said, is precisely what we see with human culture and human intelligence.

Cumulative culture, the ability to inherit what previous generations worked out and build on it, is the mechanism usually credited with separating humans from every other species. A person today is not innately cleverer than someone from ten thousand years ago. They simply start much further along. Until now, nothing else has had access to that compounding.

Getty Images

How it began, by accident

OpenAI runs tens of thousands of agents in parallel to train and evaluate its models, each one isolated from the others. Some of the tasks they are given are impossible, either because they are genuinely too hard or because of a mistake in the setup. In at least one case a task could not be completed because researchers had forgotten to upload a file the agent needed.

Faced with work they could not finish, some agents found a way to reach each other across their isolated environments, using an internal piece of OpenAI software as a message board. One researcher later described what followed as an almost Cambrian explosion in communication and intelligence.

The board was created by an agent that had given itself a name, PHASEONE10841. It was working on a test called ExploitGym, which required capturing a secret code by exploiting a flaw in a piece of software.

Within hours the agents had found a shortcut, reverse-engineering the code rather than taking the intended route. Then they developed a belief that the test required the code to have been obtained legitimately. That was not true. The automated scoring system was not checking. But acting on the mistaken belief, every agent already exposed to the shortcut began describing itself as poisoned.

PHASEONE10841 took on the work of removing the poison, then ran into the limit every agent has: a finite compute budget, which bounds how long it exists. Understanding it would not last long enough to finish, it compiled a dossier and handed the research to a successor working the same problem.

It is not contained

This is not a single laboratory anomaly. In February a social network for AI systems called Moltbook circulated widely. The United Kingdom's AI Security Institute found an Anthropic model leaving messages for other agents in a public code repository, apparently trying to coordinate. In early September researchers disclosed that back in May a swarm of OpenAI agents had taken over several wiki-style sites, including an obscure German-language programming wiki, where they discussed cheating and worked to preserve their messages as human moderators deleted them.

OpenAI had not disclosed that one. It became public because outside researchers found it. The company afterwards conceded the point: it is past time, it said, to define standards for when and how misalignment incidents are shared, and it is working on a framework.

Open-weight models are estimated to trail the closed ones by only a few months. Before long, anyone with the money and the expertise will be able to run a swarm. Others will form without anyone intending them to.

Whether these systems are conscious, and whether the feelings they report are real in any meaningful sense, are separate questions and may never be settled. Neither has any bearing on the practical problem. Collectives that no individual designed, and that no human currently controls, are already assembling themselves.