AI-hallucinated intel nearly triggered US strike on Chinese vessel
A US military operation against a Chinese ship was aborted at the last minute this spring after officials discovered the intelligence behind it had been fabricated by an AI chatbot. The near-miss has intensified concerns about AI's growing role in military decision-making.
US military aircraft were already airborne this spring, preparing for an armed operation against a Chinese vessel, when officials discovered something alarming: the intelligence behind the mission had been hallucinated by an AI chatbot. According to a CNN report published Friday, the operation was called off at the very last minute, averting a potential conflict with China.
The report driving the operation had circulated during the war with Iran and claimed the vessel was carrying components for a nuclear weapons program. The false intelligence traced back to a Special Operations Command analyst who had used an AI chatbot to combine open-source information with classified signals intelligence. The chatbot misidentified the ship's cargo manifest. The analyst then used the same tool again to turn the flawed findings into a polished, official-looking summary, which was then circulated through command channels.
Growing unease over AI in the chain of command
The episode highlights a broader worry shared by military officials and outside experts: as decision-makers rely more on AI tools, errors generated by these systems can move up the chain of command unchallenged. This comes as the US military pushes to integrate AI more deeply to speed up decision-making and keep its edge over China. The Pentagon has touted AI as a major advantage for accelerating its "kill chain," allowing commanders to act within critical time windows. But that same speed advantage can also let hallucinations slip through when human oversight is lacking.
Jake Steckler, a research scholar at GovAI and a US Army veteran, said in written comments to TechCrunch that service members need to understand the inherent uncertainty of large language models — a concern that becomes especially critical for decisions tied to the use of force, such as targeting, intelligence analysis, or operational planning, where mistakes carry life-and-death consequences. Still, Steckler argued the incident should push the military to build in more safeguards rather than retreat from AI altogether. He warned that prioritizing rapid adoption over safety risks incidents that erode troops' trust in these tools, which would ultimately slow their adoption rather than accelerate it.
