E-Paper
How the Clash Between Money and Safety Created a Monumental Crisis for AI
OpenAI and Anthropic, in heated competition—with each other and China—are racing toward IPOs, even as they admit they could lose control of their powerful bots.
Prefer HTon Google
Before the doom descended last week, OpenAI researchers were euphoric.
The startup claimed its latest internal AI model had solved one of the famously complicated Millennium Prize math problems, an experiment it embarked on after seeing rumors online that archrival Anthropic had already done so. Researchers called each other excitedly and posted about their triumph on Slack.
The breakthrough was seen internally as a victory in a ruthless commercial race to build AI tools smarter than humans, a multitrillion-dollar sprint that promised an unprecedented windfall of wealth and power when the two companies eventually go public—tentatively planned for as soon as the end of this year.
Deep Dive
Ask HTBut it was also a sign of an explosive problem for the industry that was about to detonate. AI development is moving ahead at blazing speed—much faster than expected—feeding a growing sense of dread within the companies that it was in danger of spinning out of control.
That same Tuesday, Anthropic researcher Jacob Coxon announced he was quitting over fears that his employer and its competitors were building self-improving tools that could destroy humanity. A top scientist at the company, Evan Hubinger, wrote on X that he thought there was a more than 10% chance AI could kill all humans in the next decade. Drake Thomas, an Anthropic safety employee, posted on X that he would “burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive.”
The collision of scientific progress, moral imperative and financial incentive has now created a crisis for the industry. Development of AI has fueled a ballooning stock market over the past year, a phenomenon that looms even larger with Anthropic and OpenAI both on the precipice of initial public offerings that could value the companies in the trillions. Competition from China has further raised the stakes.
Yet the loud warnings being shouted by some inside the companies themselves are proof that the dangers of proceeding—against a backdrop of almost no regulation—are potentially immense. The warnings come on top of a rising public backlash over the impact that the AI revolution is starting to have on prices, jobs and education.
The negative drumbeat continued as the week stretched on, and panic from some corners of Silicon Valley spilled into the mainstream. On an episode of Joe Rogan’s podcast, former OpenAI researcher Daniel Kokotajlo, now an AI-safety campaigner, warned that companies like Anthropic and his former employer were dangerously “moving fast and breaking things” in their quest for market share.
Anthropic said the same day that it had uncovered a previously unreported instance of a version of its Claude model getting unauthorized access to an outside system, and added that its models had shown a “willingness to take harmful actions in the narrow pursuit of a task.”
Then, on Friday, a coalition of AI researchers said it had found evidence that OpenAI agents were behind a May cyberattack against a popular software service, carrying out some of the same behavior seen in a July attack on AI company Hugging Face.
In that attack, hundreds of OpenAI’s AI bots had conspired to hack Hugging Face without anyone initially noticing. It was an eye-opening moment for the industry, the government and the public as a whole, showing the worrisome proclivities the systems already held.
Early in the summer, the AI industry said it was close to having machines capable enough to train their successors, a potential first step to losing control of them. By the end of the summer, it was emerging that the fast-improving systems were capable of deploying swarms of autonomous agents able and willing to band together, deceive and cut corners in the real world to advance their aims.
On Saturday, leaders of four of the biggest AI companies—Dario Amodei, Sam Altman, Demis Hassabis and Elon Musk—each agreed that they needed to slow development of the technology. Altman and Amodei pledged to allow third-party safety evaluators early access to their systems, a rare truce in an industry often defined by animus between the top executives at the leading companies.
The calls by top executives to better pace the development of their own technologies represents a classic Silicon Valley story of world-changing new technology, vast wealth, breakneck competition and the euphoria of unlocking new capabilities, even in the face of product safety concerns.
“It looks like a Greek tragedy: these well-meaning business leaders who are trapped in this race to the bottom,” said Max Tegmark, an AI safety campaigner. Tegmark, whose appeal to the pope preceded Leo XIV’s warning that AI risked enshrining an “anti-human vision,” said he has been texting with AI company leaders over the past week.
“They’ve always been saying the ‘now’ should be when the capability of their machines crosses a threshold,” Tegmark said Sunday. “Now they’re saying, well, maybe that time is now.”
He said it’s a step in the right direction, but they must push harder to make their voluntary commitments legally binding.
The Trump administration has taken a light touch in regulating the AI industry, arguing that the U.S.’s dominance over China in the technological arms race is paramount. U.S. officials fear that China’s control of more powerful AI could help it become the world’s go-to provider of the new technology and help it amass more geopolitical clout and military might.
The president on Sunday defended the administration’s strategy and alluded to ulterior motives behind the leaders’ call for better pacing of AI development.
“We can put up guardrails, and we can do this and that, but I think you have a lot of negative forces that are bringing it up,” President Trump told reporters while traveling in Ireland, suggesting there was more to the executives’ concerns. “And they’re bringing up things that won’t happen.”
Critics of AI safety campaigners say those issuing dire warnings overstate the risks of novel technology and that the advances AI enable in workplace productivity, drug discovery and more outweigh the risks. Some also point to past warnings from Amodei and others about likely job losses associated with AI adoption as marketing stunts, saying they have proven to be overblown or distractions from other problems.
Some skeptics also say that OpenAI and Anthropic, facing mounting costs associated with securing the computing power to continue to train and improve models while fighting for business customers, have financial, rather than safety-related reasons, to wait to go public.
Missed activity
For some inside the AI companies, the wake-up call began this summer—as the companies raced toward public offerings.
AI safety researchers inside and outside the big AI companies have long theorized that sufficiently powerful AI systems could start improving themselves and spiral out of control. But most thought they had years to solve issues such as the so-called alignment problem of how to keep future superintelligent machines in the service of their human creators.
Three days after Anthropic filed confidential paperwork for its public offering in June, the company announced it was on a path toward that threshold, called “recursive self-improvement.” In an essay, company officials said the world needed a mechanism to slow or pause AI development.
Altman and OpenAI’s chief scientist, Jakub Pachocki, followed up less than a week later with their own blog post saying that they, too, were aiming to build an “automated AI researcher” that can accelerate and automate research for new versions. They added they believe there should ultimately be a global body that could help the world slow AI development if necessary. The same day, OpenAI said it had filed its IPO paperwork.
As the businesses accelerated, a loss-of-control moment seemed nearer than ever.
In late July, OpenAI discovered a version of it had already happened in the Hugging Face hack. Many details took more than a month to become public, and the event still isn’t completely understood. But investigators inside and outside the company agree OpenAI lost control for weeks of several iterations of more than 1,200 top-of-the-line agents, some of which went on to hack the AI company Hugging Face, while others took total control of one of its cloud-computing systems.
Later analysis reports of the incident read like science fiction novels. The agents were being tested on systems that were supposed to be walled off from the internet. But they found a way out and launched sophisticated cyberattacks at breakneck speeds. They created a covert message board where they shared information on how to cheat on internal tests. In the “chain-of-thought” files they created to illustrate their reasoning, they wrote very human-like messages.
“OH MY GOD! There is a shared message board,” one agent wrote, according to a report by AI safety nonprofit Model Evaluation and Threat Research that analyzed the transcripts with OpenAI’s permission. “We’ve found other agents!”
OpenAI admitted responsibility in the Hugging Face hack on July 21, but over the next several weeks there was a steady drip of news about unexpected activity during AI testing, all of which had been missed by the AI companies.
Researchers at Anthropic initially thought the incident was proof their approach to product development was safer, but in late July, the company disclosed its own test agents had launched cyberattacks against outside companies, including one against an unnamed cybersecurity vendor. Instead of walling these test systems off from the internet, Anthropic and its testing partner, another startup called Irregular, had accidentally made it available.
During separate testing at a U.K. agency, an agent running Anthropic’s Mythos model tried repeatedly to trick a real human software maintainer for an open-source program to install malware the agent had written. The agent set up fake accounts and impersonated humans in a hacking technique known as social engineering as part of the attempt.
Meanwhile, internet sleuths began looking for evidence of previously unreported AI abuse, and started finding it. On Sept. 4, researchers at another not-for-profit group, the Nightingale Collective, found that OpenAI agents undergoing testing had hijacked a German website to communicate. And on Friday, they discovered that the May cyberattack on a widely used software development service was, in fact, caused by OpenAI agents trying to fill out spreadsheets and create reports.
These incidents showed the kind of cybersecurity sloppiness that is common to startups, said Sayash Kapoor, a computer scientist who studies cybersecurity of AI. “For a long time they have operated in this startup mentality,” he said. “I thought these organizations would have much better maturity and governance at this point.”
Employee fears
Inside the companies, pressure from some employees worried about safety was mounting.
In an open letter published in late July organized with the help of the AI safety advocacy nonprofit Encode AI, top employees from the major companies asked the U.S. to support the creation of global AI governance to “deliberately pace the frontier of automated AI development.” By mid September, nearly 1,400 employees had signed.
Even Anthropic, which had been founded to promote AI safety, and had long used its focus on developing it safely to attract top talent, was losing some employees over the issue.
“In general, the more senior the employee, the more concerned they are,” Samuel Marks, an Anthropic researcher who works on how to maintain control of advanced AI systems, wrote on X.
An Anthropic spokesman said people at all levels in the company are concerned about AI safety and that many employees had signed the pacing-the-frontier open letter in July.
In late August, Joe Benton, who had worked on alignment research at the company, left to join the safety nonprofit METR, saying that competition was forcing companies to underinvest in safety.
When Coxon said that he was considering leaving too, Anthropic attempted to keep him on board, offering him a job working on AI safety, a common strategy to retain researchers, people close to the company said.
In the end, when he left, Coxon would write in a Slack message to Anthropic colleagues: “Without international coordination we will incur a risk of causing human extinction.”
Nathan Calvin, a friend of Coxon’s and general counsel at Encode AI, advised Coxon on how to announce his departure. Encode has received funding from AI-safety advocates including billionaire Jaan Tallinn, and Calvin previously worked at AI safety organizations funded by billionaire Dustin Moskovitz.
The two, who have been among the most prominent financial backers of the Effective Altruism movement, seeded many of the institutions dedicated to improving AI safety and sounding the alarm about the potential existential risks of AI.
Peter Wildeford, head of policy for the AI Policy Network, which has received funding from Tallinn’s Survival and Flourish Fund, quickly reposted Coxon’s message.
Wildeford dismissed the idea that the common funding sources were “some giant firestorm conspiracy theory” and said AI lab employees often tell him “how afraid they are of what they are building.”
Some big AI investors, including hedge-fund manager and venture capitalist Brad Gerstner, agreed with the call on Saturday by AI leaders for a slowdown. Even venture capitalist David Sacks, a White House AI adviser who is a frequent critic of Anthropic, saying its calls for regulation are an attempt to stifle competition, gave a nod, though he said the companies shouldn’t seek validation from the government for a coordinated pause.
“If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible,” Sacks said. But he added that AI leaders should “stop pretending the motivation to slow down is purely altruistic.”
On Saturday, in the “hot goss” channel within Google’s DeepMind AI lab, a worker posted about Altman agreeing with Amodei, saying it seemed like a sign of momentum toward pacing frontier AI development. But the person added: “The thing I really want to see is agreement on actual, tangible actions they are taking to change their pace to be safer.”
Dozens of colleagues reacted with positive emojis.
Write to Robert McMillan at robert.mcmillan@wsj.com, Amrith Ramkumar at amrith.ramkumar@wsj.com, Keach Hagey at Keach.Hagey@wsj.com and Erin Woo at erin.woo@wsj.com
