The US military swung into action to intercept a Chinese ship in West Asia earlier this year, which, according to an intelligence report circulated at the time, was transporting components of a nuclear weapons program.
According to a report by CNN, citing sources, plans were put into motion for armed US military personnel to board the ship and military planes were already in the air.
All of this was based on an intelligence report circulated this spring, while the US-Iran war had started over Washington’s vow not to let Tehran have nuclear weapons. The only problem was that the report, prepared by a special operations command analyst, had been generated using artificial intelligence (AI).
According to sources quoted in the report, a chatbot used by the analyst inaccurately identified the material the ship was carrying.
The report was “entirely false”, a source told CNN, but it almost started a war between the US and China, as any operation against a Chinese vessel could have escalated into a military conflict between the two biggest economies in the world.
How AI misled US intel
According to the report, the analyst queried a chatbot about some intelligence reporting on the ship’s manifest that originated with US Special Operations Command Pacific, based in Hawaii. It was unclear whether the chatbot was a commercially available product or a US government product.
{{/usCountry}}According to the report, the analyst queried a chatbot about some intelligence reporting on the ship’s manifest that originated with US Special Operations Command Pacific, based in Hawaii. It was unclear whether the chatbot was a commercially available product or a US government product.
{{/usCountry}}“The internal tools are mostly just copies of the commercial stuff wearing lipstick,” a former senior US official familiar with the AI systems used by military and intelligence analysts told CNN.
The bot fused together open-source intelligence with secret signals intelligence in government holdings and reached its fateful conclusion about the material the ship was carrying.
The analyst then used AI again to package the findings into a standard intelligence report and sent it out.
Neither the US Special Operations Command Pacific nor the Pentagon has issued any official statement on the issue.
Use of AI in the US intelligence apparatus
Since the Donald Trump administration came to power in January last year, it has pushed for faster AI integration in the US intelligence apparatus. The reason given: It can help the military make battlefield decisions, like which targets to strike or which military assets to move where, faster.
In January, US defence secretary Pete Hegseth released his agency’s “Artificial Intelligence Acceleration Strategy” in a bid to speed up the military’s use of AI.
“We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI,” Hegseth said in a speech announcing the strategy.
The strategy also pushes for its broad use across the military, ordering the department to make AI available via several programs with the aim of “democratizing AI experimentation and transformation across the Department by putting America’s world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels,” a memo announcing the strategy said.
But the issue is, according to intelligence experts quoted in the report, that different parts of the government use different tools under different orders and safety standards. There’s no one set of standards for how the US verifies the information generated by these tools.
If true, the episode shows the profound risks of using the powerful but new and relatively poorly understood technology for targeting in the middle of a war. Analysts have long feared that AI could lead to a catastrophic miscalculation if countries are relying on poor or corrupted data, the kind of miscalculation that might lead the United States to fire on a Chinese ship based on inaccurate information.