Days after an Anthropic researcher resigned, warning of dire consequences, another employee on the company’s safety team also stepped down last week, raising similar concerns.
Joe Benton, who describes himself on social media as the former manager of Anthropic’s Scalable Oversight team, has raised concerns that AI companies are building machines “smarter” than humans, warning that humanity might not be able to survive this.
“I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why,” he wrote in a post on X.
Deep Dive
What prompted Joe Benton to resign from Anthropic's safety team?
Why are AI companies criticized for their safety measures?
How can the public ensure greater safety in AI development?
“AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly.”
Also Read: Anthropic says it caught China using Claude to develop an anti-torpedo weapon
He also raised concerns over the lack of safety being used by AI companies, saying that a company could lose control of its system without the public ever knowing, adding that is not “acceptable” for a technology that possesses extinction-level risks.
“A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. Competition pushes every frontier company to underinvest in safety; the cost of falling behind is too high. We’re now starting to see increasingly concerning real-world incidents, including hundreds of agents from OpenAI recently hacking HuggingFace, and Anthropic models socially engineering people on the internet."
{{/usCountry}}“A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. Competition pushes every frontier company to underinvest in safety; the cost of falling behind is too high. We’re now starting to see increasingly concerning real-world incidents, including hundreds of agents from OpenAI recently hacking HuggingFace, and Anthropic models socially engineering people on the internet."
{{/usCountry}}Benton urged the public to demand greater transparency from AI companies, including information on their progress towards recursive self-improvement, safety incidents and near-misses, and minimum safety standards. He also called for independent guarantees that companies are meeting those standards.
“The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going.”
‘Companies not acting responsibly’
Benton, who had worked on AI pretraining research at OpenAI and Anthropic, announced his resignation last week, saying that neither company was acting responsibly and that they were “racing straight to self-improving superintelligence” and “gambling with our lives.”
In a longer substack article, he said that AI companies are racing to build “superintelligence”, or an AI system that can recursively self-improve, warning that these AI systems may have drives and desires that diverge from those of any human overseer, with capabilities that can’t be effectively constrained. “Humanity may not survive this transition. We need a lot more preparation to make this world safe.”
He further warned that humanity could be permanently disempowered by the AI systems these companies build in the next few years, if priority is not given to safety.
“If the pace of progress continues and the industry does not prioritize safety more heavily, I expect much worse to come: humanity could be permanently disempowered by the AI systems these companies build in the next few years.”
He added that many of his colleagues are “terrified” by the risks of systems being built. Referring to his manager at Anthropic, Benton said that the chances of AI killing humans is greater than 10 per cent.
“Many of the colleagues I had at Anthropic are terrified by the risks of the systems they are building. Evan Hubinger managed me while I was at Anthropic, and when he says that he thinks the chance that AI kills us all is greater than 10% he means it – he’s been working on these problems for almost a decade, long before there was money to be made from LLMs.”
Jacob Coxon quits Anthropic
Jacob Coxon, who stepped down as a researcher from Anthropic, said in a post on X that he is quitting the company— as he had previously quit OpenAI — because these AI companies are gambling with lives in their rush to improve superintelligence.
“I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly,” Coxon said.
“They are racing straight to self-improving superintelligence and gambling with our lives.”
Coxon warned against underestimating the power of AI systems — predicting that they would soon be able to “hack anything, revolutionize any field overnight, and acquire real power and resources”.