The headlines were grandiose and practically wrote themselves: "AI Passes Medical Licensing Exam!" "ChatGPT Will See You Now" Tech companies and their supporters proclaimed a breakthrough—artificial intelligence had conquered one of medicine's most challenging gatekeepers, the board examinations.
In our lab, we had a different reaction: So what?
Here's an uncomfortable truth the AI hype machine doesn't want you to hear: The USMLE and other standardized tests measure a small fraction of what makes someone a physician. The rest is where AI falls apart spectacularly.
The Test Doesn't Test What Matters
Let's be clear about what passing the USMLE actually means. It's a test designed to assess whether a medical school graduate has memorized enough facts to begin learning how to be a doctor. Not to practice independently. Not to make life-or-death decisions. Simply to start the real training.
We've developed a fixation on standardized testing in medicine, as if multiple-choice questions could capture the essence of clinical competence. But here's what these exams can't measure:
How you respond when a patient codes at 3 AM
Whether you can feel the difference between a benign and malignant lymph node
Your ability to recognize when a mother's "by the way" comment reveals the real reason for the visit
How you navigate delivering bad news to a family
The judgment to know when to change your practice because this particular patient needs something different than what’s in the textbook
AI passing the USMLE is like memorizing cookbook recipes and claiming you can run a restaurant.
Miller's Pyramid: Where AI Gets Stuck
Medical educators use something called Miller's pyramid to visualize clinical competence. At the bottom: "Knows" and "Knows How"—the stuff you can test with multiple choice questions. These are where AI shines.
But actual medical practice happens at the top: "Shows How" and "Does." This is where you perform procedures, make decisions under uncertainty, and take responsibility for outcomes. This is where AI doesn't just struggle—it simply cannot participate.
Consider what happens during medical training:
A resident learns to intubate not by reading about airway anatomy but by feeling the resistance of tissue, adjusting to unexpected anatomy, managing their own adrenaline
An intern develops clinical intuition through sleepless nights, second-guessing decisions, carrying the weight of near-misses
A fellow masters not just surgical technique but the judgment of when NOT to operate
Which standardized test measures any of this?
The Six Competencies AI Can't Fake
The Accreditation Council for Graduate Medical Education doesn't assess residents with multiple-choice tests. They require demonstration of six core competencies—and AI can barely manage one:
Medical Knowledge: Yes, AI can retrieve and synthesize medical information. Congratulations, it's Wikipedia with better search.
Patient Care: Can AI perform a cardiac exam? Feel for organomegaly? Notice that subtle change in skin turgor that signals dehydration? Physical examination isn't just data collection—it's the cultivation of sensory intelligence that takes years to develop.
Professionalism: When AI makes an error that harms a patient, who loses their license? Who faces the family? Professional accountability requires moral agency, something no algorithm possesses.
Interpersonal Skills: We tested this. AI can generate sympathetic-sounding text. But can it read the room? Notice when silence is more therapeutic than words? Recognize that a patient's anger about wait times is really fear about their diagnosis?
Practice-Based Learning: Medical training is built on learning from mistakes—your mistakes. The sleepless night after a missed diagnosis, the visceral memory of a complication, the personal responsibility that transforms error into expertise. AI updates parameters; physicians carry scars.
Systems-Based Practice: Medicine isn't just biology—it's navigating insurance appeals, coordinating with social services, understanding which attending physician to call at 2 AM. It's knowing that Mrs. Garcia can't afford Eliquis but has a pharmacy that can get her warfarin for free.
The Dangerous Distraction
Here's what really concerns us: The obsession with AI passing standardized tests isn't just wrong—it's stifling further healthcare innovation.
While we're busy debating whether AI can replace physicians (it can't), we're ignoring what AI could actually do: eliminate the administrative burden that has physicians spending over half of their time on paperwork instead of patients.
Physicians are burning out at record rates, not because diagnosing disease is too hard, but because they're drowning in documentation, prior authorizations, and quality metrics. They're leaving medicine not because AI threatens their diagnostic skills, but because they spend more time with screens than with sick people.
We don't need AI to pass more tests. We need AI to do the work while doctors care for patients.
Beyond the Testing Trap
Something we've been discussing in our lab meetings: What if we stopped measuring AI against human tests entirely? The stethoscope didn't need to pass a hearing exam. The X-ray machine didn't take anatomy boards. They found their value by augmenting human capabilities, not mimicking them.
The real breakthrough won't come when AI scores higher on the USMLE. It will come when:
Physicians spend all of their time with patients and interacting with colleagues
Medical errors decrease not because AI makes diagnoses but because it catches possible errors or prompts protocol reminders
Burnout plummets because doctors can focus on patient care and complex decision-making while AI handles routine documentation
Patients get both cutting-edge clinical care AND genuine human connection
The Test That Actually Matters
Want to know if AI is ready to practice medicine? Don't give it a standardized test. Put it in a room with a scared patient. Have it deliver a terminal diagnosis to a young parent. Ask it to decide whether to continue aggressive treatment or transition to comfort care. Require it to carry the weight of its decisions home at night.
Until AI can do that—until it can engage with the full spectrum of what medicine demands—let's stop pretending that test scores mean anything at all.
The USMLE measures whether you're ready to start learning medicine. After passing it, AI—like any medical school graduate—would need years of supervised training to become competent. The difference? Human trainees can actually complete that training. AI can't even begin.
Maybe instead of celebrating AI's test scores, we should focus on something more urgent: using AI to give human physicians back the time to practice the medicine that no standardized test can measure—the kind that heals.
For our complete analysis of why AI medical education is impossible, read our full commentary in AEM Education & Training.
In our meetings at the Human Experience and Advancement Lab, we examine where healthcare technology hype meets human reality. This piece reflects ongoing discussions about medical education, AI capabilities, and what we're really measuring when we measure medical competence.