Carolina stood on stage at OpenAI's product launch, speaking with the measured confidence of someone who had once navigated the unimaginable. Cancer at 39. Her story was compelling: she had survived multiple types of cancer, including an aggressive form that required complex treatment decisions. ChatGPT had translated incomprehensible medical reports into plain language, helped her weigh treatment options, and ultimately empowered her to advocate for herself.
The audience nodded approvingly as she described feeling "equipped" and "informed" for the first time in her medical journey. It was a perfect Silicon Valley healthcare narrative—technology democratizing medical knowledge, AI leveling the playing field between patient and physician.
But as I listened to colleagues in our lab dissect this presentation, I realized we might be witnessing something far more complex than a simple empowerment story. We might be creating new problems faster than we're solving old ones.
The Information We Already Have
The OpenAI demonstration tapped into something fundamental about healthcare anxiety: our desperate desire for answers in a system that often trades in uncertainty. When faced with medical jargon and competing specialist opinions, who wouldn't want an AI that could provide "thorough" analysis "within minutes"?
But here’s what the polished presentation didn't address: we already live in an era of unprecedented medical information access, and this hasn't made us healthier. For nearly two decades, patients have had internet access to the same knowledge repositories that inform these AI systems. Yet we haven't seen corresponding improvements in health outcomes or patient satisfaction.
Research consistently shows that patients who engage in extensive online health research actually experience increasedanxiety and healthcare utilization compared to those who rely primarily on physician guidance. This is healthcare's "information paradox". More knowledge doesn't automatically translate to better decisions or outcomes.
What We're Seeing in Practice
A colleague recently described a scenario that's becoming increasingly common in our emergency department: patients arriving with AI-generated lists of tests they believe they need, having received algorithmic reassurance that they should "advocate for specific interventions."
In one case, a patient with ongoing discomfort had consulted AI about next steps after initial testing came back normal. The AI's response? "You still have options. There are things you can do to get the care you believe you need."
This puts clinicians in an impossible position. They must either order additional tests to address AI-generated concerns—which are potentially unnecessary and expensive workups—or spend precious consultation time explaining why algorithmic recommendations don't apply to specific clinical contexts. In this form, the chatbot incentivizes neither evidence-based care nor meaningful patient-provider dialogue.
This isn't entirely new. WebMD didn't reduce healthcare visits—it increased them. Patients armed with differential diagnoses demanded workups for rare conditions. The "worried well" became the "diagnostically worried." Now imagine this effect amplified by AI that can provide more sophisticated analysis and craft more compelling arguments for testing.
The Liability Question Nobody's Answering
The cancer patient's story reveals another troubling dimension. When her oncologists disagreed about treatment options—a reasonable clinical scenario where patient autonomy should drive decision-making—she felt "unprepared" to choose. So she turned to ChatGPT to help her make the decision.
We've somehow evolved from rejecting physician paternalism to embracing algorithmic paternalism. Patients seemingly want autonomy from doctors but dependence on AI.
In American medicine, responsibility means liability. When complications arise, someone gets blamed. The shift toward patient autonomy was partly designed to distribute this risk appropriately. But if patients increasingly rely on AI for medical decisions, who bears responsibility when things go wrong?
The regulatory framework for AI medical advice remains unclear, creating a liability vacuum that could harm both patients and providers. The FDA's current software-as-a-medical-device pathway wasn't designed for AI systems that learn and evolve over time.
Why More Information Doesn't Help
Perhaps most concerning is the fundamental assumption underlying these AI health tools: that better information leads to better decisions. But anyone who has worked in emergency medicine knows this isn't how human decision-making actually works.
Patients don't seek unnecessary interventions because they lack information about evidence-based guidelines. They seek them because they're anxious, in pain, or influenced by personal experiences and cultural expectations. People make medical decisions based on fear, relationships, cultural beliefs, and intuition—not algorithmic analysis.
Recent encounters illustrate this complexity. When presented with evidence-based options for managing a health concern, some patients respond with confusion: "Why are you asking me what to do? You're the doctor." Others arrive with detailed AI-generated plans, expecting providers to implement them regardless of clinical appropriateness. Neither scenario represents ideal shared decision-making.
This aligns with research from behavioral economics showing that access to more options actually leads people to feel less satisfied with their choices—the paradox of choice in action.
The Utilization Tsunami
The most predictable consequence of widespread AI health advice? Massive increases in healthcare utilization. If every patient can generate a personalized list of "tests you should ask your doctor about," what happens to an already strained healthcare system?
Early data supports this concern. Not only do symptom checking apps lead to higher resource utilization, when used at the point of triage, AI tools suggested greater testing than their human counterparts performed. While AI companies tout these tools as reducing healthcare costs, the evidence suggests they may actually increase utilization and costs.
Armed with AI-generated hypotheses, every patient becomes their own medical researcher. As a result, physicians are forced to validate these claims through testing or spend precious time refuting them.
What Actually Works
This isn't an argument against patient education or technological innovation. AI does excel at certain functions, specifically translation and synthesis. It can make medical jargon comprehensible and help patients formulate better questions for their doctors. Studies show AI-assisted patient education tools can improve understanding of complex medical procedures by nearly 30%.
But AI fails at wisdom. It cannot replace the clinical judgment that comes from examining patients, understanding their social context, and navigating the profound uncertainties that define medical practice. It cannot shoulder the weight of life-altering decisions, despite its willingness to try.
The most promising AI applications in healthcare focus on supporting rather than replacing clinical decision-making. Systems integrated into electronic health records improve efficiency and reduce administrative burden for providers. AI-assisted triage systems in emergency departments have shown potential to reduce wait times while maintaining diagnostic accuracy, but only when they operate within existing clinical workflows, not as standalone patient-facing tools.
The Research We Need
Instead of celebrating individual success stories, we need systematic studies comparing patient outcomes before and after widespread AI health tool adoption. We need to measure not just patient satisfaction scores, but actual clinical endpoints: appropriate healthcare utilization, diagnostic accuracy, treatment adherence, and long-term health outcomes.
We need to understand how these tools affect healthcare equity. Do they primarily benefit well-resourced patients who already navigate the system successfully? Do they create a two-tiered system of AI-informed versus AI-uninformed patients?
Most importantly, we need to study systemic effects. What happens to physician-patient relationships when algorithms become the third party in every medical conversation? How do we maintain trust and therapeutic alliances when patients increasingly arrive with AI-generated agendas?
The National Institutes of Health has recognized this need, launching research initiatives to study AI's impact on patient outcomes and health care delivery. But we need these answers before, not after, widespread deployment.
The Path Forward
The cancer patient's story is genuinely moving, and her positive outcome is encouraging. But policy decisions shouldn't be based on individual anecdotes, no matter how compelling. We've learned this lesson repeatedly in medicine: dramatic case reports make poor foundations for clinical practice.
The real question isn't whether AI can help individual patients feel more informed. Clearly it can. The question is whether widespread deployment of these tools will improve population health outcomes or simply create new forms of healthcare dysfunction.
Until we have that evidence, perhaps the most appropriate response to AI healthcare empowerment stories is the same skepticism we apply to any other medical intervention: show us the data.
Because in medicine, good intentions and gripping narratives are never enough. The road to healthcare hell is paved with technologies that helped to solve problems we thought we understood, only to create new ones that were never anticipated.
And those new problems? They're usually a lot harder to solve than the original ones we set out to fix.
References
1. Experiences with Web Search on Medical Concerns and Self Diagnosis https://pmc.ncbi.nlm.nih.gov/articles/PMC2815378/
2. Artificial Intelligence in Optimizing the Functioning of Emergency Departments; a Systematic Review of Current Solutions https://pmc.ncbi.nlm.nih.gov/articles/PMC10988184/
3. The Paradox of Choice https://thedecisionlab.com/reference-guide/economics/the-paradox-of-choice
4. AI-Enabled Medical Devices https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices
5. AI in Healthcare and the Liability Vacuum in EU Law https://healthlawsweden.blogg.lu.se/2025/05/05/ai-in-healthcare-and-the-liability-vacuum-in-eu-law/
6. Patient wisdom should be incorporated into health AI to avoid algorithmic paternalism https://pubmed.ncbi.nlm.nih.gov/36823303/