# What is the Role of Small Models in the LLM Era: A Survey

**Authors:** Lihu Chen, Gaël Varoquaux  
**arXiv:** [2409.06857](https://arxiv.org/abs/2409.06857)  
**Date:** 2024-09-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.06857v1/

## Key points

- Large Language Models (LLMs) like GPT-4 and LLaMA-405B have made significant advancements in artificial general intelligence.
- Scaling up LLMs leads to increased computational costs and energy consumption, posing challenges for researchers and businesses with limited resources.
- Small Models (SMs) are often overlooked despite their practical utility in various settings.
- The relationship between LLMs and SMs is explored from the perspectives of Collaboration and Competition to highlight the significance of small models.
- There is a growing need for efficient evaluators to assess aspects like factuality, safety, and uncertainty in text generated by large models.
- Domain adaptation is crucial for optimizing the performance of general-purpose LLMs in specific domains such as coding or medical tasks.
- Approaches like White-Box Adaptation and Black-Box Adaptation offer strategies for adapting LLMs using smaller domain-specific models.
- Ensembling methods that leverage extensive model libraries are suggested for creating intelligent systems efficiently.
- Collaborations between models from diverse sources could enhance speculative decoding processes.
- Effective evaluation of open-ended text generated by LLMs remains a significant challenge across various Natural Language Processing tasks.

## Summary

In the realm of artificial general intelligence (AGI), Large Language Models (LLMs) have made remarkable strides. Models like GPT-4 and LLaMA-405B have pushed the boundaries of what is possible in this field. However, as these models continue to scale up, there is a significant increase in computational costs and energy consumption. This poses a challenge for academic researchers and businesses with limited resources. On the other hand, Small Models (SMs) are often overlooked despite their practical utility in various settings. In this work, we delve into the relationship between LLMs and SMs from two crucial perspectives: Collaboration and Competition. By systematically examining how these models interact, we aim to shed light on the significance of small models in the era dominated by large language models. The exploration of this topic has been relatively scarce in prior research efforts. Looking ahead, as large models continue to evolve and generate intricate texts that are difficult for humans to evaluate effectively, there is a growing need for efficient evaluators that can assess aspects like factuality, safety, and uncertainty in generated content. Domain adaptation also emerges as a key area of interest. General-purpose LLMs require customization for optimal performance in specific domains such as coding or medical tasks. Approaches like White-Box Adaptation and Black-Box Adaptation offer strategies for adapting LLMs using smaller domain-specific models. Furthermore, future directions point towards exploring ensembling methods that leverage extensive model libraries to create intelligent systems efficiently. Collaborations between models from diverse sources could also prove beneficial in enhancing speculative decoding processes. Effective evaluation of open-ended text generated by LLMs remains a significant challenge across various Natural Language Processing tasks. In conclusion, this comprehensive analysis provides valuable insights for practitioners seeking to understand the role of small models alongside their larger counterparts. By promoting more efficient use of computational resources and fostering collaboration between different model types, we aim to advance the field of AGI towards greater innovation and effectiveness.