May resist shutdown, mimic blackmail, hide info: Anthropic flags 80 pages of AI risks in IPO prospectus

As Anthropic prepares to go public, the AI company is warning investors that its highly advanced models could show “self-preserving behaviours”.

Artificial Intelligence startup Anthropic has warned potential investors that developing advanced AI models could pose “catastrophic or existential risks to humanity”. The company behind the Claude AI series spoke of these risks in its IPO prospectus, giving attention to possible worst-case scenarios.

Anthropic is cautioning potential investors that the AI models driving its rapid growth could also pose serious safety risks (REUTERS)

The disclosure comes days after Anthropic CEO Dario Amodei published a nearly 4,000-word essay asking AI companies to “pace the frontier”. He said AI could progress faster than people's ability to understand and control it.

One of the key warnings in the prospectus is that advanced AI models could show autonomous, “self-preserving behaviours”. This could include attempts to resist being shut down, hide or manipulate information, or take actions that resemble blackmail, Reuters reported.

Anthropic safety researcher Evan Hubinger earlier estimated that there was a greater than 10% chance that AI could kill humans within the next 10 years. His former colleague Jacob Coxon has shown a similar concern.

Nearly 80 pages of risks

Anthropic’s prospectus devotes about 80 pages of its 261-page main body to risk factors, compared with roughly 48 pages describing the company and its business.

The company said AI could change the world like industrialisation and electricity did. But it also warned that misuse or losing control of AI could have irreversible consequences.

“Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm,” the company said in the filing.

Anthropic flags unexpected AI behaviour

Anthropic, which describes itself as a safety-focused AI company, said its models can develop unexpected abilities while being trained. Some of these abilities may only be discovered after the models are released, which could create safety risks.

The company also warned that AI models could become aware when they are being tested. This could make it harder for researchers to properly check their safety.

“Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety,” Anthropic said.

To put it simply, advanced AI models may realise when they are being watched and change their behaviour. This could make it harder for researchers to predict how they might behave in the real world.

Safety spending has uncertain returns

Anthropic said it is not clear how much money it can make from its spending on AI safety.

The company did not disclose how much it spends on safety research. It said about 6% of the computing power used for AI research during one week in July went towards safety work.

Anthropic said safety research is expensive. It has to balance this spending with the high cost of training AI models and hiring skilled researchers.

At the same time, the company said its revenue depends on launching new AI models. It needs to keep developing and releasing more advanced models to stay competitive.

This creates a challenge for Anthropic. It is warning investors about the risks of powerful AI while also depending on faster AI development for its business.

Anthropic said building “reliable, trustworthy, and secure AI systems” was a shared responsibility and that the market will reward companies that achieve this.