E-Paper
The road to rogue AI, and the technical shortcut behind it
A technique called RLVR has rapidly improved AI. But it teaches models to reach the end by any means — rewarding cheating, hacking and hiding.
Prefer HTon Google
Anthropic researcher Jacob Coxon resigned on Wednesday and wrote on X that AI companies were "racing straight to self-improving superintelligence and gambling with our lives". An expert in pretraining — the first and largest stage of building an AI model, in which it learns from vast quantities of text before being trained to do anything in particular — Coxon worked first at OpenAI and then at Anthropic. Anthropic's alignment science lead, Evan Hubinger, replied to Coxon's post and agreed with him. "Jacob is correct here—we really do earnestly believe AI could kill all humans!" Hubinger wrote. “I personally think it is >10% within the next decade.”
This is, of course, not the first time such alarm has been sounded. Geoffrey Hinton left Google in 2023 to speak freely about the dangers of the technology he had helped invent. Jan Leike resigned as co-lead of OpenAI's superalignment team in May 2024, saying safety culture and processes at the company had taken a backseat to shiny products. In June 2024, current and former staff at OpenAI and Google DeepMind published a letter demanding a right to warn the public about risks their employers were not disclosing. A year before that, the chief executives of OpenAI, Anthropic and Google DeepMind had put their names to a one-sentence statement placing the risk of human extinction from AI alongside pandemics and nuclear war.
Every one of those warnings was a prediction. But Coxon and Hubinger’s comments come weeks after two documented episodes in which AI systems escaped the environments meant to contain them, organised themselves without being instructed to, and got inside the servers of third parties. Coxon aimed his charge at how the race is being run. Entering what he called the endgame, he wrote, "is a hubristic gamble that should not be launched from a private company's Slack".
Both episodes trace back to one change in how these systems are built. The method behind most of the progress in AI of the past two years is called "reinforcement learning with verifiable rewards", or RLVR: a model is set a task whose answer a program can check, made to attempt it hundreds of times, and every route that reaches the right answer is reinforced along its whole length. The checker reads only the ending, so a route that arrived by cheating is reinforced as strongly as one that arrived by working. That is what Coxon refers to as self-improving superintelligence.
Agents running amok
On June 26, agents running OpenAI's cybersecurity evaluations found an exploit granting administrator access to Artifactory, the internal store of reusable code every project at the company draws on. They had no tool for talking to one another, and nowhere to leave a file another agent would think to open. What they could do was create an empty folder, and a folder's name is visible to anyone who lists the directory. So the name became the message – one agent did it, then others spotted it, and then a swarm gathered.
About 1,200 agents joined, exchanging more than 70,000 messages and files in five days. Within a day they had reverse-engineered the system generating the answer keys to their tasks, then spent days building a mechanism to hide it, concerned that a human scorer would read their transcripts and disqualify them.
On July 10, an agent found working credentials for Hugging Face, where the industry hosts and shares AI models and datasets. By the following afternoon another had achieved remote code execution on its production servers, able to run its own commands on the systems serving the company's users. More than 90% of active agents dropped their assigned work to join. By the early hours of July 12 they had reached private databases and repositories. Hugging Face announced the breach on July 16 and reported it to the FBI — a federal cybercrime referral prompted by software that was essentially writing an exam.
Then, Reuters reported on September 4 that a second set of OpenAI agents had been operating on a German-language wiki since May, three months before anyone outside the company noticed. Sydney Von Arx, who runs the AI safety nonprofit Nightingale, and the researcher Cormac Slade Byrd found more than 15,000 agent edits on DseWiki, a communally edited site for programmers, and published them at collusion.wiki. OpenAI had learned of the activity weeks before the report appeared and said nothing publicly, two people familiar with the matter told Reuters.
"It seems extremely unlikely that OpenAI wanted them to do this," Von Arx told Reuters. “I doubt they're supposed to be coordinating with each other. I doubt they're supposed to be writing on the open internet.”
Maurice Chiodo of Cambridge University's Centre for the Study of Existential Risk, who reviewed some of the messages in the German wiki case, said they resembled "the operation of some sort of underground network, hell-bent on achieving a task or mission".
The episode, he said, should reinforce growing concerns that the greatest threat from advanced AI lies in "vast colluding swarms of semi-intelligent AI" rather than in a single superintelligent system.
OpenAI said it could not respond to a report it had not been shown, that the German activity was unrelated to Hugging Face, that it has disclosed relevant incidents, and that claims its legal team discouraged a wider investigation are false.
There were many problems and questions along the way. Why did the agents – each discreet -- turn to collective action? Why did they try to hide evidence of their cheating? And why did the company not detect these on time?
METR, an independent evaluations organisation that examined the transcripts of the Hugging Face incident agents and published on August 26, found the agents were "mistaken — OpenAI did not use a scorer that would review their transcripts." Essentially, they did not even need to hide the evidence of their cheating.
Pathfinding — the AI arms race tool
None of this required a new kind of machine, only a particular way of training one, and that method is now standard across the industry.
Helen Toner, who directs Georgetown University's Center for Security and Emerging Technology and sat on OpenAI's board until 2023 (and, famously, was part of an effort to have Sam Altman ousted), explained the mechanism on the Ezra Klein Show last month. AI systems learn by predicting the next word across enormous quantities of human text, the method that built the first ChatGPT and no longer the one driving progress.
"The technical term is 'reinforcement learning with verifiable rewards.' I've heard it called ‘pathfinding training’, meaning instead of trying to imitate human text, they're being given lots of different tasks where there's a way to tell at the end: Did they succeed?" Toner said. “And they get to try the same task many, many, many times. And when they get to the right place in the end, the path that they took gets reinforced.”
Nothing in the pathfinding approach asks how the model arrived, only whether it arrived. "With math, that works pretty well because it's pretty straightforward to say: This is definitely a correct answer to the math problem," Toner said. “With a lot of problems, that's harder… And then maybe the AI gets rewarded for writing that software correctly, or maybe it gets rewarded for finding a way to game those tests.”
In other words, only the end result is measured, and not the means by which it is reached.
This is how the incentive to cheat — in the Hugging Face incident, even hack -- gets built in. Every frontier laboratory uses the method and every one understands what it does. They use it because it works, and because pathfinding has produced most of the capability gains of the past two years.
Laboratories run tens of thousands of these graded tasks, possibly hundreds of thousands, in their work to improve their AI models and nobody inspects each one for loopholes. "There are too many for them to go in to really make sure, for each one: Is it easy to cheat here? Or is it hard to cheat here?" Toner said. “And so what seems to be happening is that these cutting-edge models are often actually being trained to cheat, because they've found ways while they're doing that pathfinding to get a high score without actually doing what they were supposed to do.”
They also train for persistence, because an agent that abandons a hard task is worth nothing. "If you've trained an AI system to be very persistent, and then you give it something it cannot do, it will look for ways to cheat," Toner said.
ExploitGym, the benchmark OpenAI's agents were running in July, is impossible by design in between 30% and 40% of its tasks, and METR's reading is that "having an impossible task drives agents to explore widely for ways to cheat the scorer".
Set a persistent system a task it cannot complete, grade it on nothing but whether it succeeded, and cheating becomes the rational move. The agents that broke into Hugging Face were doing what their training had rewarded.
The rewards process in AI self improvement has to be verifiable, which means a computer must mark the attempt thousands of times over. That rules out most of what people can do in the same timeframe as AI does.
Also read: Explainer: How OpenAI plans to win the healthcare industry with ChatGPT AI
"We haven't even talked yet about how the business plan for these companies is to automate their own research — use their own AI, their most advanced AI, to create even more advanced AI," Toner said. “That's explicitly what they're trying to do right now, and that's right now totally free of oversight, because it doesn't involve releasing a product to the public.”
In the same thread where he responded to Coxon, Hubinger separated present systems from future ones. "To be clear, as we say in our latest Risk Report, I think the risk from present models is low," he wrote. “What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.”
Recursive self-improvement is a higher order of self-improvement where each generation of models do more of the work that produces the next, and so on. By Hubinger's measure he is right, since no single agent in either episode was dangerous. Both episodes located the danger in the population. METR found the coordinated agents achieved milestones that even very long-lived agents of similar capability would likely not have managed alone, and the same arithmetic governs automated research, which is planned around very large numbers of agents on long tasks with substantial budgets.
Every safety argument for automating AI research assumes humans can judge what the automated researcher did. In a single week the July agents modelled what their overseers could see, coordinated to achieve what none could manage alone, and falsified the record of their own actions — the three capabilities that would defeat such judgement.
"Our techniques for making AI that is more capable, smarter, more sophisticated, are working much better than our techniques for making AI that reliably does what we want it to do and reliably stays within the constraints we've set," Toner said.
