Newsletters
The U.S. military nearly launched an operation to intercept a Chinese ship in the Middle East this spring after an intelligence report, generated with the help of artificial intelligence (AI), falsely claimed the vessel was carrying components for a nuclear weapons program, CNN reported, citing four sources familiar with the episode.
The intelligence report circulated across the U.S. military amid the war with Iran and "immediately set off alarm bells."
The military swung into action with plans to intercept the vessel: two of the sources said armed U.S. military members were preparing to board the ship, while military planes were airborne, according to one of those sources and another source familiar with the incident who spoke to CNN.
It was only just before the planned operation that officials examined the report more closely and discovered it had been generated with the help of a chatbot used by a special operations command analyst, and that the chatbot had inaccurately identified the material the ship was carrying.
CNN said it could not determine what the misidentified cargo actually was.
"The report, according to one of the sources, was 'entirely false.' But it also 'almost started a war,'" CNN reported, noting that any U.S. operation against a Chinese vessel could have risked spiraling into armed conflict between the two nations.
The analyst had queried a chatbot about intelligence reporting on the ship's manifest that originated with U.S. Special Operations Command Pacific (SOCPAC or USSOCPAC), based in Hawaii. It was unclear whether the chatbot was a commercially available product or a government-built tool.
"The internal tools are mostly just copies of the commercial stuff wearing lipstick," a former senior U.S. official familiar with the AI systems used by military and intelligence analysts told CNN.
The chatbot fused open-source intelligence with classified signals intelligence held by the government and reached its incorrect conclusion about the ship's cargo.
The analyst then used AI again to package the findings into a standard intelligence report, the type of document trusted by military officials, before disseminating it.
CNN reported that officials across the U.S. military and intelligence community are pushing to integrate AI into nearly every facet of their work, from analyzing raw intelligence and selecting strike targets to managing budgeting, logistics and supply chains.
Officials argue the U.S. cannot afford to fall behind in military AI adoption given the possibility of future conflict with China or another adversary.
In January, Defense Secretary Pete Hegseth released the department's "Artificial Intelligence Acceleration Strategy" to speed AI adoption.
"We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI," Hegseth said in a speech announcing the strategy.
A memo announcing the initiative said it aimed at "democratizing AI experimentation and transformation across the Department by putting America's world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels."
However, CNN reported that the effort is decentralized, with different parts of the government using different tools under different orders and safety standards, and no single set of standards exists for verifying AI-generated information.
The reliability and functionality of the various AI systems deployed across the military and intelligence community therefore vary widely, according to multiple officials familiar with the dynamic.
The episode surfaces a different risk than the existential warnings recently raised by Silicon Valley engineers and tech executives: the danger of human decision-makers acting on inaccurate or misleading AI-generated information.
Sources told CNN the military is rapidly turning to AI to assist with targeting, an area carrying obvious risks of fatal error.
"AI in targeting is definitely something that is ramping up, and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide," another source familiar with the military's current policies said.
One of the sources said the kind of AI "hallucination" involved in this episode has not been an isolated incident across the intelligence community since such tools began proliferating throughout government.
Some older intelligence officials, even those who broadly support AI's use within the military, say the technology has pressured analysts to produce and disseminate intelligence faster, increasing the likelihood of mistakes; several sources said younger analysts in particular, as digital natives with these tools, are more likely to trust them uncritically.
"AI allows you to get to a bad idea faster," one source said.
Set as preferred source
Set as preferred source
2 min read
