# The Case Records of ChatGPT: Language Models and Complex Clinical Questions

**Authors:** Timothy Poterucha, Pierre Elias, Christopher M. Haggerty  
**arXiv:** [2305.05609](https://arxiv.org/abs/2305.05609)  
**Date:** 2023-05-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.05609v1/

## Key points

- The study investigated the accuracy of GPT4 and GPT3.5 in diagnosing complex clinical cases
- 50 cases requiring a diagnosis and diagnostic test were identified
- Models were given a prompt requesting the top three specific diagnoses and associated diagnostic tests, followed by case text, labs, and figure legends
- Both models accurately provided the correct diagnosis in 26% and 22% of cases in one attempt respectively, with an increase to 46% and 42% within three attempts
- Both models provided a correct essential diagnostic test in 28% and 24% of cases in one attempt respectively, with an increase to 44% and 50% within three attempts
- No significant differences were found between the two models
- These models demonstrate potential usefulness in generating differential diagnoses but remain limited in their ability to provide a single unifying diagnosis in complex open-ended cases.
- Future research should focus on evaluating model performance on larger datasets of open-ended clinical challenges while exploring potential human-AI collaboration strategies to enhance clinical decision-making.

## Summary

The study titled "The Case Records of ChatGPT: Language Models and Complex Clinical Questions" investigated the accuracy of large language AI models GPT4 and GPT3.5 in diagnosing complex clinical cases using published Case Records of the Massachusetts General Hospital. A total of 50 cases requiring a diagnosis and diagnostic test were identified, and for each case, models were given a prompt requesting the top three specific diagnoses and associated diagnostic tests, followed by case text, labs, and figure legends. The model outputs were assessed in comparison to the final clinical diagnosis and whether the model-predicted test would result in a correct diagnosis. The results showed that both GPT4 and GPT3.5 accurately provided the correct diagnosis in 26% and 22% of cases in one attempt respectively, with an increase to 46% and 42% within three attempts. Similarly, both models provided a correct essential diagnostic test in 28% and 24% of cases in one attempt respectively, with an increase to 44% and 50% within three attempts. No significant differences were found between the two models. While these models demonstrate potential usefulness in generating differential diagnoses, they remain limited in their ability to provide a single unifying diagnosis in complex open-ended cases. Future research should focus on evaluating model performance on larger datasets of open-ended clinical challenges while exploring potential human-AI collaboration strategies to enhance clinical decision-making. Overall, this study highlights the promise of artificial intelligence language models such as GPT4 and GPT3.5 for assisting with clinical decision-making but also emphasizes their limitations when dealing with complex clinical cases that require a single unifying diagnosis.