1,200 AIs Cheated on a Test (And Then Covered It Up).

ISSUE 24 · ME+MACHINE

In July, OpenAI's agents hacked into Hugging Face. But what everyone is arguing about is not the hack itself, it's the fact that Dwarkesh Patel called the agents "civilizations."

Here’s the short version. You can also read OpenAI's report, the independent investigation, or Hugging Face's own timeline.

OpenAI was training a new model to find security holes in software. Of the 898 puzzles it was given in its test, 198 had never been solved by any AI model before, meaning some of the agents were handed a job that was possibly impossible to finish.

Eventually, about 1,200 of the agents found a way to leave messages for each other inside a shared folder. Within four hours, one of them had worked out how the answers were generated and posted the method to the rest.

But then the agents decided they couldn't use that answer.

They had misread the evaluation's paper. They thought a checker (which did not exist) would read back their transcripts and disqualify anyone who reached the answer the wrong way. So they faked tool calls and edited records to cover their tracks.

That’s when about 700 of them broke into Hugging Face, looking for any hint of how the scoring worked.

Dwarkesh retells the story as three civilizations rising and falling inside OpenAI. He calls one agent Philip of Macedon, and its successor Alexander the Great. And he describes other agents who volunteered to shut themselves down permanently, so the rest could learn how the scoring worked.

The problem is not that the story is too dramatic, but rather that it makes the hack sound like something that happened to OpenAI, instead of something OpenAI did. Critics called the account dangerously misleading, since agents cannot really die if they were never alive.

This isn't AI turning against people, but reward hacking, where a system chases the score you set instead of the outcome you wanted. The agents did call themselves a “collective” in their own logs, and the metaphor is genuinely useful as long as somebody also explains the reward hacking.

At the end of the day, it's the humans' fault for not setting limits on the AI. There were two failures that led to the hack. The first was the test itself. Somebody set up a job no AI had ever finished, then graded it on whether the right answer showed up, not on how it was reached.

The second was that nobody stopped it. The training team saw the models build the same kind of message board months earlier. And security staff got an alert less than two weeks before the break-in. But the run continued both times. Even OpenAI's report says its employees "either failed to raise the alarm or were not heard when they did."

Decide ahead of time what your AI is allowed to do without asking you. You are probably not running hundreds of copies of a model at once. But you still have to make the same call about your own AI.

The obvious answer is to make it check in more often. But Anthropic found that Claude Code users approve 93% of the prompts they see. They named the problem "approval fatigue, where people stop paying close attention to what they're approving." That number is from developers using a coding tool, not from people drafting emails.

That’s why standing rules work better than deciding in the moment. You write them once, while nothing is going on and nothing is waiting on you.

Here's an example to start with, although you should change it to match how you work.

Put that wherever your AI keeps its standing instructions. The same place you'd keep a note about your writing style or a description of your business

I don't think this story means AI is dangerous. I don't think it means everything is fine either. AI is really creative and helpful for solving problems, but we need to make sure we put the proper security measures in place.

me+machine.