# Combining Insights From Multiple Large Language Models Improves Diagnostic Accuracy

**Authors:** Gioele Barabucci, Victor Shia, Eugene Chu, Benjamin Harack, Nathan Fu  
**arXiv:** [2402.08806](https://arxiv.org/abs/2402.08806)  
**Date:** 2024-02-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.08806v1/

## Key points

- Study explores potential of large language models (LLMs) in improving diagnostic accuracy in medical settings
- Existing LLMs may not always exhibit necessary level of accuracy for real-life applications
- Researchers employed collective intelligence methods and analyzed dataset of 200 clinical vignettes
- Combining responses from multiple diverse LLMs led to significantly higher accuracy in differential diagnoses compared to relying on single LLM outputs
- Average accuracy for aggregated responses from three LLMs was $75.3\%\pm 1.6pp$, while single LLM-generated diagnoses had an average accuracy of $59.0\%\pm 6.1pp$
- Utilizing collective intelligence methods enhances diagnostic accuracy by integrating insights from various LLMs, reducing reliance on a single commercial vendor
- Benefits of leveraging multiple sources of AI-driven information to enhance decision-making processes in healthcare settings

## Summary

In their study titled "Combining Insights From Multiple Large Language Models Improves Diagnostic Accuracy," authors Gioele Barabucci, Victor Shia, Eugene Chu, Benjamin Harack, and Nathan Fu explore the potential of large language models (LLMs) in improving diagnostic accuracy in medical settings. LLMs like OpenAI's GPT-4 and Google's PaLM 2 have been touted as valuable tools for diagnostic support or even replacements for traditional consults. However, existing LLMs may not always exhibit the necessary level of accuracy required for real-life applications, even when specifically trained on medical topics. To address this limitation, the researchers employed collective intelligence methods and analyzed a dataset comprising 200 clinical vignettes of real-life cases. They compared the accuracy of differential diagnoses generated by individual commercial LLMs (including OpenAI GPT-4, Google PaLM 2, Cohere Command, and Meta Llama 2) against those synthesized by aggregating responses from combinations of these LLMs. The results indicated that combining responses from multiple diverse LLMs led to significantly higher accuracy in differential diagnoses compared to relying on single LLM outputs. Specifically, the average accuracy for aggregated responses from three LLMs was found to be $75.3\%\pm 1.6pp$, whereas single LLM-generated diagnoses exhibited an average accuracy of $59.0\%\pm 6.1pp$. Through their findings, the authors highlight the effectiveness of utilizing collective intelligence methods to enhance diagnostic accuracy by integrating insights from various LLMs. This approach not only demonstrates improved diagnostic performance but also reduces reliance on a single commercial vendor—a crucial step towards establishing broader acceptance and utility of LLMs as reliable diagnostic support tools in medical practice. Overall, this research underscores the potential benefits of leveraging multiple sources of AI-driven information to enhance decision-making processes in healthcare settings and emphasizes the importance of collaborative approaches in optimizing diagnostic outcomes using advanced language models.