# Sequential Diagnosis with Language Models

**Authors:** Harsha Nori, Mayank Daswani, Christopher Kelly, Scott Lundberg, Marco Tulio Ribeiro, Marc Wilson, Xiaoxuan Liu, Viknesh Sounderajah, Jonathan Carlson, Matthew P Lungren, Bay Gross, Peter Hames, Mustafa Suleyman, Dominic King, Eric Horvitz  
**arXiv:** [2506.22405](https://arxiv.org/abs/2506.22405)  
**Date:** 2025-06-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2506.22405v2/

## Key points

- AI has the potential to revolutionize access to expert medical knowledge and reasoning in healthcare.
- Traditional evaluations of language models often fall short in capturing the complexity and nuance of evidence-based medicine in real-world scenarios.
- Researchers have introduced the Sequential Diagnosis Benchmark to bridge the gap between AI capabilities and clinical reality by transforming diagnostically challenging cases into stepwise diagnostic encounters.
- The MAI Diagnostic Orchestrator (MAI-DxO) adds sophistication to the diagnostic process by simulating a panel of physicians, suggesting likely differential diagnoses, and strategically selecting high-value, cost-effective tests.
- MAI-DxO achieves an impressive 80% diagnostic accuracy rate when combined with OpenAI's o3 model, significantly outperforming generalist physicians and traditional AI models.
- MAI-DxO reduces diagnostic costs by 20% compared to traditional physician-led approaches and by 70% compared to off-the-shelf AI models like o3.
- When configured for maximum accuracy, MAI-DxO achieves an outstanding 85.5% accuracy rate across various AI model families such as OpenAI, Gemini, Claude, Grok, DeepSeek, and Llama.
- The research highlights how AI systems can enhance diagnostic precision while improving cost-effectiveness in clinical care settings.

## Summary

Artificial intelligence (AI) has the potential to revolutionize access to expert medical knowledge and reasoning in healthcare. Traditional evaluations of language models often fall short in capturing the complexity and nuance of evidence-based medicine in real-world scenarios. In clinical practice, physicians engage in an iterative process of formulating and revising diagnostic hypotheses, adapting their questions and tests based on new information, and carefully weighing evolving evidence before arriving at a final diagnosis. To bridge this gap between AI capabilities and clinical reality, researchers have introduced the Sequential Diagnosis Benchmark. This innovative approach transforms 304 diagnostically challenging cases from the New England Journal of Medicine clinicopathological conference into stepwise diagnostic encounters. In this model, a physician or AI begins with a brief case abstract and must iteratively request additional details from a gatekeeper model that reveals findings only upon explicit inquiry. Performance is evaluated not just based on diagnostic accuracy but also on the cost-effectiveness of physician visits and tests conducted. Furthermore, the introduction of the MAI Diagnostic Orchestrator (MAI-DxO) adds another layer of sophistication to the diagnostic process. This model-agnostic orchestrator simulates a panel of physicians, suggesting likely differential diagnoses and strategically selecting high-value, cost-effective tests. When combined with OpenAI's o3 model, MAI-DxO achieves an impressive 80% diagnostic accuracy rate—four times higher than the average performance of generalist physicians. Moreover, MAI-DxO significantly reduces diagnostic costs by 20% compared to traditional physician-led approaches and by 70% compared to off-the-shelf AI models like o3. Notably, when configured for maximum accuracy, MAI-DxO achieves an outstanding 85.5% accuracy rate across various AI model families such as OpenAI, Gemini, Claude, Grok, DeepSeek, and Llama. These remarkable performance gains underscore how AI systems can enhance diagnostic precision while simultaneously improving cost-effectiveness in clinical care settings. In conclusion, this research highlights the transformative potential of AI-guided iterative thinking and judicious decision-making in advancing healthcare diagnostics towards more accurate outcomes at reduced costs.