# Large Language Models: A Survey

**Authors:** Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, Jianfeng Gao  
**arXiv:** [2402.06196](https://arxiv.org/abs/2402.06196)  
**Date:** 2024-02-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.06196v1/

## Key points

- Thorough review of Large Language Models (LLMs) and their advancements
- Current state of LLMs and focus on three prominent families: GPT, LLaMA, and PaLM
- Overview of early pre-trained neural language models: NLMs based on RNNs (LSTM and GRU) and Transformer architecture
- Use of self-attention mechanisms in Transformers for efficient pre-training of large language models
- Techniques used to enhance LLMs for real-world applications
- Survey of popular datasets prepared for LLM training, fine-tuning, and evaluation
- Review of widely used evaluation metrics for LLMs
- Comparison of performance among several popular models on representative benchmarks
- Discussion of open challenges and future research directions in the field of LLMs

## Summary

In this paper, the authors provide a thorough review of Large Language Models (LLMs) and their advancements. They begin by discussing the current state of LLMs and focus on three prominent families: GPT, LLaMA, and PaLM. The authors also give an overview of early pre-trained neural language models that laid the groundwork for LLMs. These include NLMs based on recurrent neural networks (RNNs), such as LSTM and GRU, as well as the revolutionary Transformer architecture. The paper then delves into how LLMs are built, highlighting the use of self-attention mechanisms in Transformers to efficiently pre-train large language models on massive amounts of data. The authors also discuss techniques used to enhance LLMs for real-world applications. Next, they survey popular datasets prepared for LLM training, fine-tuning, and evaluation. The authors review widely used evaluation metrics for LLMs and compare the performance of several popular models on representative benchmarks. Finally, they conclude by discussing open challenges and future research directions in the field of LLMs.