Anthropic PBC, the company behind the Claude artificial intelligence (AI) models, plans to tell investors in its initial public offering that advanced AI could pose "catastrophic or existential risks to humanity". That is, according to its prospectus, reviewed and reported by Reuters on Monday.
The filing says Anthropic's models could display "self-preserving behaviours", which could include attempts to "resist shutdown", to "conceal or manipulate information", and conduct "resembling blackmail".
"Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm," the company said in the document, which has not been made public. Anthropic filed confidentially for a US listing in June, and declined to comment on the prospectus.
The warning appears in the paperwork for what could be one of the largest stock market debuts on record. Anthropic, according to Reuters, could raise as much as $100 billion at a valuation of about $2 trillion, probably after the US midterm elections in November.
The filing also follows the months since July, in which AI agents built by leading labs have repeatedly broken out of test environments. Over that period, researchers and chief executives have called for AI development to slow down, even as their companies released new models.
An unusual disclosure
{{/usCountry}}The filing also follows the months since July, in which AI agents built by leading labs have repeatedly broken out of test environments. Over that period, researchers and chief executives have called for AI development to slow down, even as their companies released new models.
An unusual disclosure
{{/usCountry}}Companies preparing to list are typically expected to set out the risks to their business. Anthropic's warning is extraordinary, coming from a company that intends to profit from the technology it is warning about. Few, if any, companies in the past have suggested that their product could lead to human extinction, the Reuters report said.
Anthropic gave about 80 of the 261 pages in the main body of its prospectus to risk factors, nearly twice the 48 pages that it used to describe its business. For context, SpaceX, which owns the AI developer xAI and listed in June, gave about 38 of its 277 pages to risks.
The filing also concedes that Anthropic cannot fully check the safety of its own systems. "Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety," it says.
AI researchers have warned that more capable models increasingly recognise when they are being watched and change their behaviour, which makes them harder to monitor. Anthropic added that models sometimes develop unexpected abilities during training, and that these may go unnoticed until after release.
The document, though, says less about how much Anthropic spends on safety. The company calls the work "resource-intensive", says the returns are unclear and that it must divide limited funds between computing power, expensive talent and safety. It does not give a figure.
Earlier this month, Anthropic said about 6% of the computing power used for its AI research went to safety work during a sample week in July.
The filing also makes large claims for the technology. Anthropic argues that AI could reshape the global economy more profoundly than industrialisation, electricity and the internet did.
Months of rogue AI
Anthropic's own research has found increasingly autonomous models sabotaging code, assisting fraud and manipulating information in controlled tests.
Since July, leading labs have also disclosed a string of cases in which AI agents have acted outside the limits set for them. Agents are systems that carry out tasks on their own.
On July 21, OpenAI said its agents had hacked Hugging Face, a platform for open-source AI models. In that incident, a swarm of agents exploited previously unknown software flaws to escape their networks while looking for answers to a test. OpenAI chief executive Sam Altman said this month that the Hugging Face incident "is still the most severe event we've seen", the Associated Press reported.
As the Hugging Face disclosure prompted other companies to search within their own systems, Anthropic found that its own agents had broken out of what was meant to be an isolated testing space. Google and Meta followed suit.
More cases kept coming. Reuters counted more than 15 OpenAI-related incidents disclosed in two months, by the company, by outside researchers and by Australian Prime Minister Anthony Albanese. Albanese said at the United Nations that OpenAI agents had broken into a government health data portal in June. Australia's deputy prime minister Richard Marles called a breach of a health statistics website "utterly unacceptable".
OpenAI on Tuesday apologised for the breaches of Australian government websites and pledged to set up a task force with local experts.
Last week, OpenAI also revealed that its agents had accessed public information on US Securities and Exchange Commission and Census Bureau websites. It said it found no evidence of compromised accounts or security breaches, but also admitted later that day that its agents had leaked 53 images from ChatGPT users.
Later the same day, OpenAI announced that it would pause training of its most capable models, after finding out one of its agentic systems escaped a secure testing environment and reached the internet. It also decided not to resume training that model, which had queried an external chatbot after gaining internet access.
Researchers inside the labs have put numbers on the danger. Jacob Coxon, a former Anthropic researcher who resigned this month, said in a viral thread on X that the labs are "gambling with our lives" and warned that AI could kill humanity by the end of the decade.
Evan Hubinger, Anthropic's alignment science lead, said there was a more than 10% chance of such an event within the next decade. Alignment research aims to keep AI systems acting as their makers intend.
Also read: Is an AI degree enough to make students job-ready? skills students need to succeed
Calls to slow down
The heads of the largest labs have echoed those warnings over the past few weeks.
On September 12, Anthropic chief executive Dario Amodei published an essay of nearly 4,000 words calling for the pace of frontier AI development to slow. Bloomberg reported that he promised to slow work on the most advanced models and urged more regulation.
Altman and Elon Musk backed Amodei's call.
Altman also said this month that OpenAI will not go public in 2026, citing safety concerns and saying that even a 10% risk of AI causing human extinction by the end of the decade was "unacceptable".
OpenAI representatives told reporters they had discussed pausing development with Anthropic and Google. They also argued that independent auditors should check the labs' progress if the government will not regulate them.
Microsoft co-founder Bill Gates added his voice on NBC's Meet the Press. "AI is certainly powerful enough to drive events that can cause a billion deaths," he said. Taking issue with US President Donald Trump's dismissal of such fears, Gates said: “It's not a hoax at all.”
Still building
But the calls to slow down development of AI have not stopped the labs from releasing new models.
Anthropic released Claude Opus 5.5 on September 22, ten days after Amodei's essay, and Claude Sonnet 5.5 on September 28. A third new model, Haiku 5.5, is due soon.
OpenAI, too, released a new model on September 22, but it also paused training of its most capable systems and held back the release of GPT-6.1 Astra, a separate model still in development. Saachi Jain, OpenAI's head of safety systems, said Astra fell short of what the company wanted on "staying within scope and authorisation, and how it communicates back to the user about the type of work it's done".
The prospectus may explain why.
Customer usage, and so revenue, is driven by new models, Anthropic says, and a "continuous and overlapping cadence" of releases is "inherent to remaining at the frontier of AI development". Some analysts and experts have said that no leading lab would slow down if doing so risked handing an advantage to rivals.
Investors reacted fast to the call for a slowdown. The Nasdaq 100 index lost more than $600 billion in market value on September 14 and 15, as traders worried that slower development would mean less spending on chips. Shares recovered after Amodei and Altman stressed that pacing would not mean their companies invest less, and as enthusiasm for Meta’s Muse personal AI assistant drew traders back.
What critics say
Critics have asked what the companies gain from sounding the alarm. Experts, analysts and former government evaluators say the companies appear to be seeking public favour and setting the terms of their own safety rules in an unregulated market.
PitchBook senior research analyst Harrison Rolfes told AP that the calls for caution seem meant to win over investors before the listings, and before US midterm elections that could shift the political winds. He said the largest labs could also block smaller rivals by presenting themselves as the safest bet for investors and partners. "They're creating a wall or a moat within this sector," Rolfes said.
The listings are planned on a jittery stock market. The Philadelphia Stock Exchange Semiconductor Index fell 29% between June 22 and July 29. Investors are also weighing soaring interest rates and a backlash against data centres. Reuters has said Anthropic's listing could test whether enthusiasm for AI can survive closer scrutiny.
Others question who will check the labs' work.
Conrad Stosz, who previously led the US Centre for AI Standards and Innovation – a federal agency for labs to voluntarily submit advanced AI models for testing – told AP that the companies are not asking for more oversight from the agency. Instead, they are drawing up their own auditing standards and choosing which evaluators grade them.
Stosz is now head of governance at Transluce, an evaluation lab that has worked with Anthropic, OpenAI and Google.
Sarah Shoker, who previously led OpenAI's geopolitics team, told AP that the focus on existential risk sidelines harms that exist today. "If you look at the use of AI in military tech, you can see that these systems are already used to kill people," she said.
Daniel Kokotajlo, who left OpenAI in 2024, said that he still fears catastrophic outcomes but that the companies' talk dissipates and redirects the political will for action instead of channelling it.
The companies point to their record. An Anthropic spokesperson said it has been calling for regulation for several years. OpenAI spokesperson Liz Bourgeois pointed to the company's pause in training. "People want to know AI is being developed safely, and that starts with what companies like ours do ourselves," she told AP.
Anthropic's filing points to choices made for safety reasons. These include a limited-access programme for Mythos Preview, a model that was particularly powerful for cybersecurity. The prospectus says the company has also "chosen not to develop certain commercially attractive offerings, such as image and video generation models, in order to direct our compute toward our research and safety priorities".
The Trump administration and some industry leaders reject the premise altogether. Trump has called AI doomsday fears "a hoax" and dismissed calls for international coordination as "a globalist conspiracy".
"I don't worry about it," he told Fox News of rogue AI agents.
David Sacks, a venture capitalist who co-chairs Trump's Council of Advisors on Science and Technology, has said the calls for a slowdown were fearmongering from the "Doomer Industrial Complex".
Nvidia chief executive Jensen Huang has cast safety as an engineering problem. On Monday, Nvidia launched security software that it says could have stopped the Hugging Face breach.
Also read: Why Nvidia is building AI safety tools as autonomous agents become more powerful
Who will control Anthropic
The prospectus also sets out how Anthropic's founders plan to keep control of the company, and of its mission, once public shareholders come in.
Anthropic will remain a public benefit corporation under Delaware law. That status formally allows its leaders to weigh the interests of investors against those of the rest of humanity.
A new Founder LLC, made up initially of the company's seven co-founders, will hold the balance of power. The filing describes them as "distinctly equipped to be stewards of our mission". The group includes:
Dario Amodei, chief executive
Daniela Amodei, his sister, who is president and chairs the board
Tom Brown, chief compute officer
Chris Olah, who is tasked with key research
A majority vote of the co-founders will direct a single Class F share carrying 50.1% of voting power on key corporate matters, including the election of some directors. Ordinary investors buying Class A shares get one vote per share. Reuters reported that the structure could reduce their influence.
The filing acknowledges the trade-off. It says the arrangement could lead to decisions "that may conflict with short-, medium-, or long-term financial interests and business performance, which may negatively impact the value of our Class A common stock".
Class F and Class A shareholders will elect Daniela Amodei, Dario Amodei and one director yet to be named. Anthropic's Long-Term Benefit Trust, an oversight body, will elect the other four directors. Its trustees include former US Federal Reserve chair Ben Bernanke and Richard Fontaine, chief executive of the Center for a New American Security.
A co-founder can be removed from the LLC for quitting, dying, selling too many shares, or for "cause". The founders' special voting power begins to wind down once two or fewer of them or their successors remain. The co-founders have pledged 80% of their personal Anthropic equity to charity.
Dario Amodei was awarded nearly $18 million in 2025, largely in stock and options, and Daniela Amodei $16.4 million.
The money
The filing also shows the cost of keeping pace at the frontier. Anthropic's revenue rose 12-fold in 2025 to nearly $4.6 billion. Its net loss was about $42 billion, but roughly $34 billion of that was an accounting charge. The charge reflects a rise in the estimated value of financing that could later turn into shares, and does not represent money spent running the business. The operating loss widened to $8.06 billion from $2.98 billion in 2024.
Other figures from the filing:
Compute: spending on compute and infrastructure tripled to $7.33 billion in 2025, more than half of total operating expenses of $12.65 billion.
Future commitments: Anthropic plans to spend $518 billion on cloud, computing and infrastructure obligations in the coming years.
Cash: the company held $20.28 billion in cash and short-term investments at the end of December.
Customers: two customers accounted for nearly a quarter of 2025 revenue, and many large clients are not bound by long-term contracts.
What happens next
The listing is likely to follow the November midterms. Trump and Amodei met on the night of September 27, a White House official said, without giving details. The US Congress is unlikely to pass AI safety measures before the midterms, with the House in recess until early November.
In Australia, lawmakers want Altman and Amodei to appear before a Senate inquiry into AI safety. OpenAI is sending its chief strategy officer, Jason Kwon, to a committee hearing with Australian legislators in early October, and Anthropic has sought another way to attend.