Anthropic has urged the world's leading artificial intelligence companies to consider mechanisms to slow the development of increasingly powerful AI systems, warning that rapid advances could eventually enable models to improve themselves without human intervention and pose significant societal risks.
In a blog post published on Thursday, the company disclosed internal data showing how rapidly its most advanced models are improving.
The post cautioned that recursive self-improvement has not yet occurred and is not inevitable, “but could come sooner than most institutions are prepared for.”
Anthropic said the ability to slow global AI development would “likely be a good thing.”
Also Read | I accidentally created a monthly finance tracker with Claude AI. Here’s all about it
What is the self-improvement stage in AI?
The post, written by Marina Favaro, head of the company's internal research institute, and co-founder Jack Clark, said current advances appear to be moving toward “recursive self-improvement” - a stage where AI systems can improve themselves without human intervention. Some AI researchers view that threshold as a potential source of danger and major societal disruption.
{{/usCountry}}The post, written by Marina Favaro, head of the company's internal research institute, and co-founder Jack Clark, said current advances appear to be moving toward “recursive self-improvement” - a stage where AI systems can improve themselves without human intervention. Some AI researchers view that threshold as a potential source of danger and major societal disruption.
{{/usCountry}}“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advances of the technology,” the post says. It also proposes a global agreement on how to potentially slow development and a mechanism to verify that competitors are adhering to it.
Marketing strategy or caution?
Critics have argued that Anthropic’s warnings about the risks posed by its own tools may also serve as a marketing strategy.
They point to the company’s decision to restrict the release of its powerful “Mythos” cybersecurity model, which can identify bugs and vulnerabilities, as a way to highlight the capabilities of its products.
Anthropic’s leadership has maintained that it takes safety seriously and is encouraging broader discussions about AI risks.
Ethan Mollick, a professor at the University of Pennsylvania’s Wharton School and an influential scholar on AI transformation, said that while some critics view Anthropic’s safety messaging as publicity, many people inside the company are “true believers", according to the WSJ.
“AI labs are a mix of things,” said Mollick, whose book about AI, “Co-Existence,” is set to be released in the fall.
The AI industry has long been divided over how close current systems are to achieving milestones such as artificial general intelligence (AGI) - a level of intelligence comparable to humans - or reaching the stage of recursive self-improvement.