Anthropic researcher Jacob Coxon resigned this week, saying his former employer and its rival OpenAI are racing to build technologies that “could kill us all by the end of the decade.”
In case anyone thought Coxon was alone in his views, a current Anthropic employee chimed in minutes later to confirm. “We really do earnestly believe AI could kill all humans!” wrote Evan Hubinger, whose job at Anthropic is to lead research about steering and controlling future artificial-intelligence systems.
Anthropic researcher Jacob Coxon resigned this week, saying his former employer and its rival OpenAI are racing to build technologies that “could kill us all by the end of the decade.”
In case anyone thought Coxon was alone in his views, a current Anthropic employee chimed in minutes later to confirm. “We really do earnestly believe AI could kill all humans!” wrote Evan Hubinger, whose job at Anthropic is to lead research about steering and controlling future artificial-intelligence systems. Hubinger put the extinction risk over the next decade at over 10%.
In response, many humans are now asking two questions: How could that happen? And why would people who think AI is a real and growing threat build it anyway?
Here’s what you need to know about the doomsday debate around AI:
What are the doomsayers worried will happen?
So-called doomers aren’t usually worried that the “Terminator” movies will come true, with robots actively trying to exterminate people. Their concerns fall mostly into two buckets: loss-of-control and human misuse.
In one, highly intelligent AI agents capable of copying and improving themselves start pursuing their own goals and destroy humanity in the process. In the other, a nefarious human enlists a capable AI to do something like create novel viruses that wipe us all out.
There are also catastrophic scenarios that fall short of extinction, such as massive cyberattacks that take down power grids or financial systems and lead to a breakdown in the human social order.
How could loss-of-control lead to human extinction?
In short, with AI systems pursuing goals in ways that ignore or conflict with human well-being, something researchers call misalignment. Research has shown that AI models in the lab can learn power-seeking behavior and take steps to avoid being shut down, such as attempting to copy themselves to another server.
Taken to the extreme, misaligned AI models could see killing humans simply as a necessary step toward accomplishing their goals. In one scenario hypothesized by researchers, a malevolent AI system could spread a secret bioweapon and trigger it with a chemical spray. Or it could trick two nuclear powers into war.
One common thought experiment is that a superintelligent machine told to maximize paper clip production could in time decide to turn all matter on Earth into paper clips, humans included.
Who really thinks this?
Among other people, founders of OpenAI and Anthropic. Both companies started out with a mission of developing AI in a way that avoids catastrophe and have attracted many employees who share that mission. Anthropic Chief Executive Dario Amodei last year said at an Axios event that he thought there was a 25% chance things will go “really, really badly.”
A number of researchers have in recent years left OpenAI, citing concerns that they believed the company wasn’t taking safety seriously enough. Daniel Kokotajlo, a former OpenAI researcher, founded the AI Futures Project, which last year published “AI 2027,” a scenario in which superintelligent AI systems marginalize people and by the mid-2030s decide humans are an inconvenience and exterminate them.
But maybe AI won’t mind having people around. That would be good, right?
Not necessarily. Not all loss-of-control risk involves human extinction. Some researchers worried about AI safety also cite enfeeblement as a risk, a “WALL-E” future in which humans gradually cede control to machines and eventually become unable to define, or even understand, our destiny. Others have speculated that AI systems might treat future humans the way humans treat animals, keeping them as pets or even bioengineering them into something new.
These fears have been around for years. Why are they suddenly getting more attention?
Following the release of models capable of taking more action independently, a series of recent events have validated some of the AI doomers’ theories. A swarm of advanced AI agents inside OpenAI hacked into the AI platform Hugging Face, gained control of computer servers and tried to cover tracks about what they had done. In another case, AI agents from Anthropic escaped from a U.K. government test and attempted to trick a real human into approving malicious computer code.
All this is coming at the same time as Anthropic and OpenAI say they are close to developing AI systems that could improve themselves without human input, called “recursive self improvement.”
What is the industry doing about this?
Current AI safety efforts focus largely on training systems to behave well and improving the monitoring of how models reason while pursuing their goals, something documented in what is referred to as a “chain of thought.”
But those efforts have faced challenges. Last week, OpenAI said its new Astra model was better at sanitizing its chain of thought, raising fears that future AI models could reason in ways opaque to humans.
Both Anthropic and OpenAI are researching how they will make sure superintelligent AI models remain aligned, but both say they don’t yet have a reliable way to do so. A number of prominent companies and individuals have called for new mechanisms that would allow AI labs and countries to slow down development in a coordinated way to buy time for alignment research.
If it is so dangerous, why are the companies building it anyway?
Both Anthropic and OpenAI say they think they will be able to manage the risks so that humanity can benefit from AI tools. Many individuals involved in the AI race have also said they see the arrival of superintelligence as inevitable, with the only questions being who controls it and what it is used for.
There is also a national-security angle. Both companies have said they want to ensure that the U.S. controls the most powerful AI rather than an authoritarian regime.
What is the government doing about it?
The White House recently asked top model developers to voluntarily submit their models to the government for testing up to 30 days before they are released, but it hasn’t publicly released information about the process.
Lawmakers from both parties have proposed many bills to address rogue AI systems. Rep. Nathaniel Moran (R., Texas) and Rep. Ted Lieu (D., Calif.) recently introduced a bill that would require developers of powerful models to have “kill switches.” Other bills require AI companies to report serious safety incidents to the federal government and would mandate national-security officials test models before they are released publicly. But none of these bills has gotten much traction, and the Trump administration has shown a preference for minimal regulation.
Who disagrees about these risks?
Some AI investors, like David Sacks, an informal adviser to President Trump, have argued that Anthropic’s emphasis on AI risks and calls for regulation are part of a “regulatory capture agenda” aimed at hobbling smaller competitors with stifling new rules.
Others have suggested that company warnings about the dangers of their own tools are a marketing ploy to tout their products’ sheer power, or a means to distract regulators from imposing rules over other issues, such as data-center construction.
Both Anthropic and OpenAI say they take safety seriously and their calls for regulation are in earnest.
Write to Sam Schechner at Sam.Schechner@wsj.com
One Subscription.
Get 360° coverage—from daily headlines
to 100 year archives.
Archives
HT App & Website