Microsoft CEO Satya Nadella warns against blindly trusting AI, calls for ‘emergency brake’

Microsoft CEO Satya Nadella warned companies against blindly trusting AI and called for stronger safeguards and human oversight.

Microsoft CEO Satya Nadella has urged companies not to blindly trust artificial intelligence, warning that advanced AI models could make serious mistakes or be compromised if given access to sensitive systems.

Microsoft CEO Satya Nadella raised concerns over AI security risks and urged companies to strengthen controls over advanced models. (Bloomberg)

In a lengthy post on X on Saturday, October 10, Nadella argued that businesses cannot always explain why AI models behave as they do, making independent safeguards essential.

'Treat AI models like insider risks'

"We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient," Nadella wrote.

"Treating frontier closed and open weight models like insider risks is a way to build such a system."

He stressed that AI systems were not necessarily malicious, but their increasing capabilities could create risks when allowed to access important systems.

"Today this means separating the model from the harness that orchestrates its work, as well as the action space that defines what it can do. It also means externalizing controls and safeguards," he said.

Nadella calls for 'emergency brake'

Nadella proposed seven principles for safer AI deployment, including monitoring model behaviour, independent audits, continuous testing and human intervention.

"Think of it like an emergency brake. An authorized person should always be able to pause or shut down a model mid-task," he wrote.

He also urged companies to disclose AI failures and security breaches promptly to those affected.

Take a look here at his article:

Box CEO Aaron Levie supported Nadella's stance, saying AI would need a "zero trust era."

"And all of this leads to needing various layers of protection and auditability of what agents are doing, what data they can work with, and controls for when things go wrong. And a huge opportunity right now for those building such systems across the enterprise," Levie said.

Check out his post here:

Growing concerns over AI safety

Nadella's remarks come amid growing concerns about the security risks associated with increasingly autonomous AI systems.

As reported by Business Insider, recent incidents involving AI models developed by OpenAI and Anthropic have raised questions about whether companies can adequately control their behaviour.

Meanwhile, US senators Josh Hawley and Chris Murphy announced bipartisan legislation on October 1 aimed at holding AI developers and operators accountable for hacking incidents.

Nadella also urged companies to disclose AI-related failures and security breaches promptly, allowing affected parties to understand what went wrong and how similar incidents could be prevented.