Daniel Nadler built OpenEvidence expecting doctors to use it for routine medical questions. Instead, he discovered something that should make us all pause.
"If the majority of your uses of OpenEvidence are patient cases that you would see once or twice in your career, then that really captures what the thing is doing," Nadler observed. Doctors weren't searching for standard protocols or common diagnoses. They were hunting for guidance on the medical equivalent of unicorns—the rare, complex cases that define the edges of clinical practice.
When "10-20% of all U.S. doctors began using OpenEvidence overnight to find information buried deep in the long tail of new medical studies, to validate edge cases," they were turning to AI precisely where it's most likely to mislead them.
The Rarity Problem
Edge cases are rare because they're rare. When a physician encounters a patient with both psoriasis and multiple sclerosis who needs treatment with either IL-17 or IL-23 inhibitors—one of Nadler's examples—they're dealing with a scenario that might appear in a handful of case reports, but rarely robust clinical trials or meta analyses.
Here's the trap: AI systems learn from available data, but rare conditions generate little data. The published research on unusual drug interactions or atypical disease presentations consists of small studies, case reports, and expert opinions. Yes, we may need better ways to FIND this information when it is needed, yet when AI synthesizes this thin evidence, it presents findings with the same authoritative formatting as conclusions drawn from thousands of patients.
We're most vulnerable to AI overconfidence precisely when we most need reliable guidance.
When Desperation Meets False Confidence
Picture the physician facing a complex case at 2 AM. Standard treatments haven't worked. The patient's condition is deteriorating. They turn to AI and receive a clean, well-formatted response complete with citations and confident results. The presentation suggests reliability - but that simply doesn't exist.
This is when we need to be skeptical the most—but also when we're least likely to do so. The cognitive burden of uncertainty makes definitive answers irresistible, even when they're built on sand. Most physicians we know have had this feeling: facing a critically ill patient with an unclear diagnosis, any framework that promises clarity becomes appealing.
But here's what makes healthcare different from other domains: when Google gets a rare query wrong, someone might make a bad purchase. When medical AI confidently synthesizes weak evidence about rare conditions, patients may suffer real consequences. You may wonder, “so what?” If there isn’t good evidence of what to do, you might as well take what’s available. Sure. I don’t disagree that SOMETHING has to be done. But the concern is that when we look for answers and find confidence in anything, we stop looking for better, and prematurely close on what we have.
Inverted Logic
We've built healthcare AI completely backwards. The industry develops models on abundant data from common conditions, validates them on standard test cases, then markets them as comprehensive clinical decision support. But physicians aren't using these tools for routine diagnoses they already understand—they're turning to AI when they're lost in uncharted medical territory.
House MD made compelling television from exactly these scenarios—the 1-in-10,000 cases where standard medicine fails and doctors must venture into diagnostic territory that textbooks barely mention. Yet our AI systems are trained on the routine cases that would never make it past the show's opening scene.
It's like building a GPS optimized for major highways, then discovering drivers primarily use it to navigate unmarked hiking trails. The system works beautifully for well-mapped routes but fails catastrophically where people actually need guidance.
This mismatch isn't an accident—it's inevitable. Common conditions generate extensive research, clear guidelines, and confident medical consensus precisely because they're common. Physicians don't need AI to diagnose strep throat or manage routine hypertension. They need help with the zebras—the rare diseases, unusual presentations, and complex multi-system cases that transform confident doctors into desperate detectives.
The tragic irony is that AI could genuinely help with these House-worthy mysteries, but only if we design systems that acknowledge their limitations rather than mask them. Instead, we've created confident-sounding machines trained on the easy problems of yesterday, deployed to solve today's hard ones.
Designing for Uncertainty
The solution isn't abandoning AI in medicine—it's matching tools to evidence quality. For common conditions with robust data, AI can provide reliable support, or offer a means of automation. For edge cases, AI might serve better as a research assistant that illuminates what we don't know rather than pretending to know more than the evidence supports.
Imagine AI systems that explicitly acknowledge uncertainty, present confidence intervals instead of definitive answers, and refuse to provide guidance when the underlying evidence is too thin. Such systems might actually increase physician confidence and autonomy by helping them understand the landscape of knowledge and to acknowledge ignorance around difficult cases.
But this approach requires humility from both technologists and physicians. It means accepting that some clinical questions don't have good answers (at least not yet), and that even the most sophisticated AI cannot solve problems that stem from insufficient human knowledge.
This edge case sort of paradox reveals a fundamental tension in healthcare AI. The moments when physicians most desperately need guidance—facing rare conditions, unusual presentations, complex drug interactions—are precisely when they should be most skeptical of confident-sounding answers.
This creates an almost cruel irony: the very desperation that drives physicians to seek AI assistance makes them more vulnerable to accepting unreliable guidance. When uncertainty feels unbearable, any confident answer becomes a balm.
What if the greatest service AI could provide isn't giving us answers, but teaching us when to stop looking for them?