# Do We Still Need Clinical Language Models?

**Authors:** Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J. Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, Emily Alsentzer  
**arXiv:** [2302.08091](https://arxiv.org/abs/2302.08091)  
**Date:** 2023-02-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.08091v1/

## Key points

- Recent advances in scaling large language models (LLMs) have shown improvements in various NLP tasks.
- It is uncertain whether LLMs trained on general web text are suitable for specialized and safety-critical domains like clinical text.
- LLMs encode a significant amount of medical knowledge, raising the question of whether smaller domain-specific language models are still necessary.
- An empirical analysis of 12 different language models with varying parameters was conducted on three clinical tasks to assess their ability to parse and reason over electronic health records.
- The study did not test the models' ability to reason over long texts or consider tasks involving generating clinical text such as summarization.
- Techniques to reduce model size or perform parameter-efficient training were not explored in the study.
- Performance improvements could not be solely attributed to pretraining data distribution versus model architecture due to limitations in comparing different architectures.
- Instruction-tuned models and comparison against ChatGPT were not included due to availability constraints but should be considered in future research.
- Smaller specialized clinical language models outperformed larger domain-agnostic models even when finetuned on limited annotated data, emphasizing the importance of developing domain-specific models for highly specialized domains like clinical text.
- Further investigation is needed to address limitations, explore techniques for reducing model size and improving parameter efficiency, and compare against instruction-tuned models specifically tailored to the clinical domain.

## Summary

Recent advances in scaling large language models (LLMs) have shown improvements in various natural language processing (NLP) tasks. However, it remains uncertain whether these models, primarily trained on general web text, are suitable for highly specialized and safety-critical domains like clinical text. Recent studies have indicated that LLMs encode a significant amount of medical knowledge, raising the question of whether smaller domain-specific language models are still necessary. To address this question, we conducted an extensive empirical analysis of 12 different language models with varying parameters on three clinical tasks that assess their ability to parse and reason over electronic health records. While our study focused on evaluating the performance of these models on clinical tasks, there are several limitations and areas for future work. Firstly, we did not test the models' ability to reason over long texts, which is particularly challenging when working with clinical notes. Additionally, we did not consider tasks that involve generating clinical text such as summarization, which could pose challenges for encoder-only models. Furthermore, our work did not explore techniques to reduce model size or perform parameter-efficient training. Another limitation is that we compared different architectures without being able to attribute performance improvements solely to the pretraining data distribution versus the model architecture. We also did not include instruction-tuned models or compare against ChatGPT due to availability constraints. In future research, it would be valuable to compare against these models and develop instruction-tuned models specifically tailored to the clinical domain. In conclusion, our experiments demonstrate that smaller specialized clinical language models outperform larger domain-agnostic models even when finetuned on limited annotated data. This highlights the importance of developing models specifically designed for highly specialized domains like clinical text. Further investigation is needed to address the aforementioned limitations and explore techniques for reducing model size and improving parameter efficiency in clinical NLP tasks. We would also like to acknowledge Mark Dredze for his suggestion regarding FLOPs comparison between models and Elena Sergeeva, Geeticka Chauhan , Melina Young , and Maggie Liu for their valuable feedback and assistance in creating figures .