Three Key Takeaways
The automation trap: AI tools designed to replace doctors (rather than collaborate with them) often make clinical decisions worse, not better—even when the AI outperforms humans on paper.
Bias is structural: AI models trained on data from elite coastal academic centers won’t work well for rural populations, underserved communities, or anyone outside the “typical” patient profile in the training data.
The expertise paradox: As AI gets better at routine tasks, humans risk losing the skills needed to step in when systems fail—a lesson written in blood from aviation disasters.
I recently joined Dr. Matthew Strehlow on the Stanford Emergency Medicine Podcast to talk about where we really are with AI in emergency medicine—and where the breathless headlines diverge from the messy reality of clinical implementation. The conversation ranged from diagnostic radiology failures to aviation disasters to why your HR chatbot doesn’t need to know advanced physics.
Here’s what we discussed:
On What AI Actually Is (And Isn’t)
Me: When I think about artificial intelligence, I mostly think of it just as a way that we can use data and information technologies to mimic parts of what humans do or what we think humans use intelligence for. There’s many different ways that artificial intelligence can look like—it can be sorting through data and looking for patterns. But it can also be writing an Atlantic article or a New Yorker article, or a scientific research paper.
The reason I think it’s important to think about how we use these terms is I’ve seen papers that refer to generative AI when they’re talking about predictive AI. They mention predictive AI when they’re actually just talking about classification or sorting different data points.
The distinction matters: Don’t be overwhelmed. We’ve been using predictive analytics in medicine for decades—everything from regression analyses to liver transplant allocation algorithms. We just didn’t necessarily call it AI.
Where AI Is Being Used in EDs Today
Matt Strehlow: So where are we at now? Like what AI tools are being used in the ED today?
Me: I have been doing informatics work since my college senior thesis... and I’m gonna be honest, very little has changed. It’s still addressing the same sort of problems we see over and over again in emergency medicine.
The tools that I’ve seen people working on are things like trying to understand how people arrive in the ED and predict resources and have them available. Queuing theory is an artificial intelligence methodology to try to predict who’s gonna come in and when.
That gets you to the next logical space that everyone (I’ve seen many, many, many companies tried to work on) which is triage—improving ESI scores based on when someone comes and not leaving that up to a human who might be biased at the front desk... That still tends to be one of the biggest problems that people focus on right now in emergency medicine.
When Good AI Makes Things Worse: The CheXpert Problem
Me: In a widely discussed study published in April 2024, researchers at MIT found that when radiologists used an AI diagnostic tool called CheXpert, the accuracy of their diagnoses declined.
Even though the AI tool in the experiment performs better than two-thirds of radiologists, giving radiologists access to AI predictions does not, on average, lead to higher performance.
Why did this good tool produce bad results?
A proximate answer is that doctors didn’t know when to defer to the AI’s judgment and when to rely on their own expertise. When AI offered confident predictions, doctors frequently overrode those predictions with their own. When AI offered uncertain predictions, doctors frequently overrode their own better predictions with those supplied by the machine.
The deeper problem: This tool was designed to automate the task of diagnostic radiology—to read scans like a radiologist. But automating a radiologist’s entire diagnostic job was infeasible because CheXpert was not equipped to process the ancillary medical histories, conversations, and diagnostic data that radiologists rely on for interpreting scans.
Automation vs. Collaboration: A Critical Distinction
Me: Tools can be generally divided into two main buckets. In one bucket, you’ll find automation tools that function as closed systems that do their work without oversight—ATMs, dishwashers, electronic toll takers, and automatic transmissions.
In the second bucket you’ll find collaboration tools, such as chain saws, word processors, and stethoscopes. Unlike automation tools, collaboration tools require human engagement. They are force multipliers for human capabilities, but only if the user supplies the relevant expertise. A stethoscope is unhelpful to a layperson. A chain saw is invaluable to some, dangerous to many.
AI does not go neatly into either the automation bucket or the collaboration bucket. AI does both: It automates away expertise in some tasks and fruitfully collaborates with experts in others. But it can’t do both at the same time in the same task.
And the distinction matters because bad automation tools—machines that attempt but fail to fully automate a task—also make bad collaboration tools. They don’t merely fall short of their promise to replace human expertise, they interfere with human expertise, and sometimes undermine it.
The Air France 447 Lesson
Me: The tension between automation and collaboration lies at the heart of a notorious aviation accident that occurred in June 2009... When the pilots of Air France 447 were struggling to control their airplane, falling ten thousand feet per minute through a black sky, pilot David Robert exclaimed in desperation, “We lost all control of the airplane, we don’t understand anything, we’ve tried everything!”
At that moment, in a tragic irony, they were actually flying a perfectly good airplane. Yet the combination of startle, confusion, at least nineteen warning and caution messages, inconsistent information, and lack of recent experience hand-flying the aircraft led the crew to enter a dangerous stall. The accident report called it “total loss of cognitive control of the situation.”
Two design failures: One is that the autopilot was a poor collaboration tool. It eliminated the need for human expertise during routine flying. But when expert judgment was most needed, the autopilot abruptly handed control back to the startled crew, and flooded the zone with urgent, confusing warnings.
The second failure was that the pilots were out of practice. Human expertise has a limited shelf life. When machines provide automation, human attention wanders and capabilities decay.
On Bias in AI Systems
Matt: I’ve heard you write and speak on bias. Can you explain to people just an overview of why AI can actually exacerbate bias?
Me: There are certain hospitals that we all know that have had EHRs for a long time and early adopters—the VA is one of those, and then a lot of academic centers. When we think of academic centers and where they’re located, they’re not generally representative of a lot of the country.
Different people present to different types of hospitals and different hospitals have different access to digital versions of patient information. The data that’s been collected that we use, that we throw into the models then is from places where people have access to the highest quality of academic care on average.
San Francisco patients look a little bit different or respond a little different to people in Houston, which is a little bit different than Indianapolis, or New York City. That’s fine if you know and are aware of that, but it also means that anything being operated at scale just doesn’t have a great way of dealing with patients who go to the rural hospital that is still on a paper chart.
But there’s a flip side: I do wanna just briefly state though that it’s important to remember that humans themselves do things and make decisions that are based on our experiences and physicians may move around the country. There is a real opportunity for artificial intelligence to take out some unconscious biases that results from humans anyway.
What Keeps Me Up at Night
Matt: Tell me, what are some of the ones that really would keep you up at night? What are the biggest risks of us adopting AI in emergency care?
Me: Let’s think. The risks to whom? Let’s talk about the risk to the patients themselves and the care we provide and the risk to us as physicians.
The EHR hurt us a lot, actually caused a lot of burnout, has made it so that patients don’t feel like they get time with their doctors. So there was a harm to the patient-physician relationship because of technology.
The risk is that you automate some of that. A lot of people get a version of care that’s cookie cutter in ways and lacks the deep knowledge you need by having a human in the loop that can actually still steer the ship for them. That can leave people marginalized, at the edge, leaves the edge people even more at the edge. You have a doubling down of the negative effects for people that aren’t usually part of the health system.
But There Are Reasons for Optimism
Matt: Tell me about some other things that you think are potential positives and that you’re excited about with AI and healthcare.
Me: I worry about not setting realistic goals. I think things are blowing up and I’m actually very excited. It’s just that we are probably in the trough of disillusionment right now, because people were super hyped when they first heard of GPT but almost no business has seen a large scale return on investment.
That is a normal experience too. That is a normal, well-known, well-described Gartner hype cycle. It’s important to not become disillusioned for long-term formative change.
I think there’s a lot of reason to be worried that [our jobs] might get worse. The risk to physician practice is very real. Right now I’m sure people worry that they’re just gonna get replaced by AI tools... That has not ever been the case in automation—whether it’s tellers, autopilot in planes, the loom. It’s changed industries more than it’s ever wholesale eliminated the major players of it.
My Advice for Skeptical Physicians
Matt: So there’s a lot coming our way. What advice can you give [physicians who are feeling skeptical and overwhelmed]?
Me: Parts of innovation and informatics are all just what every emergency physician friend of ours does anyway. We process information, we try to build tools. Everyone, most of the EM physicians I meet are as good or better than me at all of these things. And they didn’t necessarily do an informatics fellowship.
That means engaging with the technology, experimenting with it wherever you can, trying stuff out to see and find when ultrasound started, ultrasound was first used, I think its first medical use was on an eyeball, because it’s a fluid-filled sac and ultrasound works really well in it, and then people started trying it on other stuff.
This is the way that medicine advances, and has been doing that since the stethoscope.
You don’t have to do it on clinical care necessarily, but you can practice and see how different models work. Free versions of almost all of these things are accessible right now before they eventually get more expensive. Ask questions, try stuff out. Think about the cases you encounter and then look to see if there’s a way for these technologies to help.
What We’re Taking Away
In our work at the HEAL Lab, we’re constantly thinking about the intersection of human experience and technology in medicine. This conversation reinforced something we’ve been discussing in recent lab meetings: collaboration beats premature automation.
The lesson from aviation, from radiology, and from decades of technology implementation in healthcare is clear. The question is whether we’ll learn it before we have our own Air France 447 moment in medicine.
The challenge ahead isn’t whether AI will transform emergency medicine—it will. The question is whether we’ll build tools that amplify human expertise or erode it. And that choice is still very much in our hands.
Listen to the full conversation on the Stanford Emergency Medicine Podcast for deeper discussion on trust, the hype cycle, and why I still dream of being an astronaut.
