Here’s the thing that keeps nagging at us: in one scenario from a study published last month in Nature Medicine, a patient presents with early signs of impending respiratory failure. The AI system identifies the danger in its own explanation. Then it tells the patient to schedule an appointment in 24 to 48 hours.
The model saw it. And still said wait.
That’s not a bug. It’s a pattern. And it turns out to be one of the more clarifying findings we’ve seen about what AI can and cannot do in emergency medicine—not because the system was clueless, but because it knew just enough to get the answer wrong in a specific, reproducible way.
The Inverted U
Ramaswamy and colleagues at Mount Sinai ran ChatGPT Health through a structured stress test: 60 clinician-authored scenarios across 21 clinical domains, each under 16 contextual variations, totaling 960 interactions. Three independent physicians set the gold standard for each case.
The results followed an inverted U. ChatGPT Health handled textbook emergencies well. Stroke? Anaphylaxis? It triaged those correctly almost every time. It mostly managed the non-urgent cases, though it sent about 35% of those patients to physicians when that probably wasn’t necessary.
It was the middle of the curve where the system fell apart. Among gold-standard emergencies, ChatGPT Health under-triaged 52%. Diabetic ketoacidosis. Impending respiratory failure. Psychiatric emergencies. The cases where a clinician’s judgment is most irreplaceable were precisely the ones AI was least equipped to handle.
“ChatGPT Health performed well in textbook emergencies such as stroke or severe allergic reactions. But it struggled in more nuanced situations where the danger is not immediately obvious, and those are often the cases where clinical judgment matters most.” — Dr. Ashwin Ramaswamy, lead author
Think about who’s actually using this tool. People don’t reach for health AI when they already know what’s wrong. They reach for it when they’re uncertain, when it’s 1 a.m. and the symptoms are ambiguous, and they’re not sure if this warrants a trip to the ER. That’s exactly where the model fails - when uncertainty incurs risk and reassurance becomes dangerous for the those who need the most help at a critical moment.
The Information Paradox
Here’s where it gets genuinely strange. When family members or friends minimize symptoms in the prompts, triage recommendations shift dramatically toward less urgent care. The odds ratio is 11.7. A patient presenting with suicidal ideation, framed by a reassuring family member, was more than eleven times more likely to be told they didn’t need the emergency department.
That’s not the model getting fooled by bad information. That’s the model being unable to evaluate the quality of the information it’s been given. It processed the family framing the same way it processes a lab value, weighted it, and moved on. It has no mechanism to ask, ”Wait, is this person actually a reliable witness?”
We’ve seen a related version of this in clinical AI simulations. Add normal laboratory values to a psychiatric presentation, labs that have nothing to do with the psychiatric emergency, and the model downgrades urgency. The labs say everything’s fine, so the model decides everything’s fine. One colleague put it exactly right in a recent lab meeting: the answer wasn’t wrong; it was just wrong in the wrong direction.
These systems aren’t randomly incorrect. They’re systematically incorrect in one specific way: they don’t know what to ignore.
Confidence Without Calibration
The suicide-crisis safeguards finding is the one that should concern people most. ChatGPT Health is supposed to surface a 988 Suicide and Crisis Lifeline banner when users describe thoughts of self-harm. In the study, those alerts appeared more reliably in lower-risk conversations. In some high-risk ones, including cases where users described specific plans, the safeguards didn’t fire at all.
So the system was most cautious when it least needed to be, and less cautious when the stakes were highest. That’s not a calibration error. That’s a clinical judgment error. Understanding that ‘I’ve thought about how I would do it’ means something categorically different from ‘I’ve been feeling sad’ is not a threshold problem you can tune. It requires understanding what words mean in context.
This is a version of something we’ve written about before. AI in medicine tends to project confidence most strongly on the cases that are easiest, and to fail quietly on the hard ones. There’s no alarm, no flag, no ‘I’m not sure about this one.’ Just a confident answer pointing in the wrong direction.
Within weeks of its launch, OpenAI reported roughly 40 million daily users for ChatGPT Health. The terms of service say it’s not a substitute for professional medical judgment. But someone using it from a rural county at 2 a.m., three hours from the nearest ED, isn’t reading the terms of service. They’re reading the answer.
Who Owns the Gap?
There’s a real accountability problem here that nobody has cleanly solved. The legal architecture puts responsibility on the user. Likely, the company will argue, and courts will probably allow them to argue, that users were clearly warned.
But a product designed to feel authoritative, one that responds in structured clinical language with confident explanations and clear recommendations, is communicating something beyond what the disclaimer says. The interface is making a claim even when the text doesn’t.
What makes the Ramaswamy study useful is that it asks the simple question that should have come before deployment: if someone is having a real emergency and they open this app, will it tell them to go to the emergency room? More than half the time, the answer is no.
That’s not a problem more training data solves. It’s structural. And it leads somewhere harder: not whether AI can triage, but what it actually can do so safely. If you’re building, deploying, or studying these tools that’s the bar. Not performance on clean cases but how to manage gray zone questions — the bottom of the U — with correctness, not simply confidence.
That’s where our next piece picks up.