Writings
AI Got the Diagnosis Right More Often Than Doctors. Here’s What That Does — and Doesn’t — Mean for You.
Originally published on Substack.
You already have a diagnosis on your phone. Or you’re about to paste your symptoms into a chat window before you call anyone. The question isn’t whether AI will show up in medicine. It’s whether you know what a high score on a written test actually buys you — and what it never can.
What happened
In a paper published in Science on October 22, 2026, a Microsoft team led by Harsha Nori and Eric Horvitz tested a system they call Diagnostic Orchestrator (DxO) — an AI that simulates a panel of physicians, revises its differential as new information arrives, and picks the next question or test with cost in mind (Nori et al., Science 2026).
They didn’t feed it a finished chart. They started with a short chief complaint, the way a real visit starts, and made the system ask for history and order tests one step at a time. On 56 New England Journal of Medicine clinicopathological conference cases held out during development, DxO paired with GPT-5 reached the right diagnosis 83.2% of the time. Active generalist physicians — primary care, hospitalists, acute and emergency physicians — scored 36.1% when they could use search and reference tools, and 22.3% closed-book. On 325 routine emergency-department admissions drawn from MIMIC-IV-ED, DxO hit 80.3% versus 46.8% for physicians (Nori et al., Science 2026).
The strategy differed too. On the hard NEJM cases, physicians asked a mean of 5.6 questions before ordering their first test. DxO asked a mean of 25.5. It spent more time on history before spending money on labs and imaging.
Why the 36% number looks worse than your last checkup
Those NEJM cases are teaching puzzles chosen because they are hard. They are not a random Tuesday in clinic. The physicians in the study worked text-only, with a gatekeeper that answered questions from a fixed case record — no body language, no family in the room, no longitudinal relationship, no specialist team. The authors say plainly that these comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice (Nori et al., Science 2026).
So the honest reading is narrower: on a controlled, text-based sequential-diagnosis benchmark, an orchestrated language model beat individual generalists on accuracy and often on estimated testing cost. It is a strong signal about reasoning under idealized information. It is not a trial of whether patients live longer or get hurt less.
What this means for you
-
☐ If a diagnosis doesn’t fit, ask “What else could this be?” — of your doctor, and of any AI you use.
-
☐ Treat chatbot answers as a differential to discuss, not a verdict to trust alone.
-
☐ Bring your full history: exposures, travel, medicines, prior workups. The AI’s edge in this study came partly from asking more history before testing.
-
☐ Ask who is accountable for the final call when AI is in the room. A score is not a license.
-
☐ Rare or “doesn’t make sense” symptoms still need a human who can examine you, order the right imaging, and own the plan.
The deeper point
Medicine has always been a conversation between what we know and what we still have to ask. Tools that force the next good question can help. Tools that replace the person who sits with you when the answer is uncertain cannot. A better differential is not the same thing as better care. Keep asking until someone who can touch your wrist writes the plan down.
Have you ever gotten a wrong diagnosis — from a person or from a machine? What finally corrected it? Tell me in the comments.
This is education, not medical advice.
Blessings,
Ash
More at DoctorEmrani.com
