# Superhuman performance of a large language model on the reasoning tasks of a physician

**Authors:** Peter G. Brodeur, Thomas A. Buckley, Zahir Kanjee, Ethan Goh, Evelyn Bin Ling, Priyank Jain, Stephanie Cabral, Raja-Elie Abdulnour, Adrian Haimovich, Jason A. Freed, Andrew Olson, Daniel J. Morgan, Jason Hom, Robert Gallo, Eric Horvitz, Jonathan Chen, Arjun K. Manrai, Adam Rodman  
**arXiv:** [2412.10849](https://arxiv.org/abs/2412.10849)  
**Date:** 2024-12-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.10849v1/

## Key points

- Large language models (LLMs) in medical tasks are typically evaluated using multiple choice question benchmarks, which may not reflect real-world clinical scenarios.
- Clinical reasoning is considered a more practical benchmark for assessing LLM performance as it involves critical thinking and synthesizing clinical data.
- Previous LLMs have shown promise in surpassing clinicians in routine and complex diagnostic scenarios.
- The study focuses on evaluating OpenAI's o1-preview model, specifically designed to enhance runtime by engaging in a chain of thought processes before generating a response.
- Five experiments were conducted to assess the performance of o1-preview: generating a differential diagnosis, displaying diagnostic reasoning, triaging differential diagnoses, engaging in probabilistic reasoning, and demonstrating management reasoning.
- Significant improvements were observed in generating a differential diagnosis and enhancing the quality of diagnostic and management reasoning with o1-preview.
- No notable improvements were seen in probabilistic reasoning or triaging differential diagnoses with o1-preview.
- There is a need for new robust benchmarks and scalable evaluations to effectively assess LLM capabilities compared to human physicians.

## Summary

, , , , 

In the realm of medical tasks, the performance of large language models (LLMs) has typically been assessed using multiple choice question benchmarks. However, these benchmarks are often limited in scope and do not necessarily reflect real-world clinical scenarios. Clinical reasoning is considered a more practical benchmark for evaluating LLM performance as it involves critical thinking and synthesizing clinical data for diagnosing and managing medical issues. Previous LLMs have shown promise in surpassing clinicians in both routine and complex diagnostic scenarios. This study focuses on evaluating OpenAI's o1-preview model, which was specifically designed to enhance runtime by engaging in a chain of thought processes before generating a response. The performance of o1-preview was assessed through five experiments: generating a differential diagnosis, displaying diagnostic reasoning, triaging differential diagnoses, engaging in probabilistic reasoning, and demonstrating management reasoning. These experiments were evaluated by physician experts with validated psychometrics. The primary objective of the study was to compare the output of o1-preview with results from identical previous experiments involving historical human controls and benchmarks from earlier LLMs. The findings revealed significant improvements in generating a differential diagnosis and enhancing the quality of diagnostic and management reasoning. However, no notable improvements were observed in probabilistic reasoning or triaging differential diagnoses. Overall, this study highlights the capability of o1-preview to excel in tasks requiring intricate critical thinking such as diagnosis and management. While its performance on probabilistic reasoning tasks aligned with past models, there is a pressing need for new robust benchmarks and scalable evaluations to effectively assess LLM capabilities compared to human physicians. Additionally, trials assessing AI applications in genuine clinical settings are essential for further validation and advancement in this field.