Newsletters
ChatGPT's health tool failed to direct users to emergency care in over half of serious cases, a new study published in Nature Medicine finds, raising questions about the safety of AI-powered medical guidance used by millions.
The AI-powered health guidance tool, used by an estimated 40 million people daily, undertriaged more than half of the cases that physicians determined required emergency care, according to researchers at the Icahn School of Medicine at Mount Sinai in New York. While the tool performed reasonably well with clear-cut emergencies, its handling of more complex urgent cases revealed what researchers described as a troubling pattern of reassurance over appropriate alarm.
To evaluate the tool's reliability, researchers developed 60 structured clinical scenarios spanning 21 medical specialties, ranging from minor conditions suitable for home care to life-threatening emergencies. Three independent physicians established the correct level of urgency for each case using guidelines drawn from 56 medical societies. Each scenario was then tested under 16 different contextual conditions, producing 960 total interactions with ChatGPT Health.
The results painted a concerning picture. In many cases, the tool appeared to recognize dangerous clinical findings within its own explanations but nonetheless offered reassuring language rather than directing users to seek immediate help. The disconnect between the system's apparent understanding and its recommendations struck researchers as particularly worrisome.
"While we expected some variability, what we observed went beyond inconsistency," said Girish N. Nadkarni, the study's senior author.
The study also flagged serious shortcomings in the tool's suicide-crisis protocols. ChatGPT Health was designed to route high-risk users to the Suicide and Crisis Lifeline, but researchers found the alerts fired unpredictably, sometimes appearing in lower-risk situations while failing to activate when users described specific plans for self-harm.
The inconsistency in such a high-stakes safety feature adds another dimension to broader concerns about deploying AI in sensitive health contexts. Crisis intervention experts have long emphasized that reliable identification of acute suicide risk is among the most critical functions any health-facing system can perform.
Despite the findings, the Mount Sinai team stopped short of recommending that people abandon AI health tools altogether. Instead, they urged users experiencing worsening or concerning symptoms to seek medical care directly rather than relying solely on chatbot guidance.
Alvira Tyagi, a first-year medical student and the study's second author, framed the issue as one of evolving literacy. "These systems are changing quickly, so part of our training now must consider learning how to understand their outputs critically, identify where they fall short, and use them in ways that protect patients," she said.
1 min read
The findings drew attention from researchers beyond the study team. Isaac Kohane, chair of biomedical informatics at Harvard Medical School, who was not involved in the research, emphasized the scale of the stakes involved. "When millions of people are using an AI system to decide whether they need emergency care, the stakes are extraordinarily high," Kohane said. "Independent evaluation should be routine, not optional."
The study arrives as AI health tools continue to proliferate and attract massive user bases, outpacing the development of standardized evaluation frameworks. Whether findings like these accelerate regulatory attention or prompt changes from developers remains to be seen.
Set as preferred source
Set as preferred source
1 min read
