# LLaMA: Open and Efficient Foundation Language Models

**Authors:** Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample  
**arXiv:** [2302.13971](https://arxiv.org/abs/2302.13971)  
**Date:** 2023-02-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.13971v1/

## Key points

- Thibaut Lavril, Giza Cardozo, Éric Grave, and Guillaume Lample introduce the LLaMA collection of foundation language models
- Models range from 7B to 65B parameters and are trained on trillions of tokens using publicly available datasets exclusively
- State-of-the-art models can be trained without proprietary and inaccessible datasets
- A smaller model trained for longer can ultimately be cheaper at inference
- The focus is to train language models that achieve the best possible performance at various inference budgets by training on more tokens than what is typically used
- LLaMA models outperform existing large language models (LLMs) such as GPT-3 on most benchmarks despite being smaller in size
- All their models are released to the research community and use only publicly available data sources for training
- Compatible with open-sourcing and democratizes access to and study of LLMs
- Modifications made to the transformer architecture (Vaswani et al., 2017) and their training method are presented
- Performance of their models compared with other LLMs on a set of standard benchmarks is reported
- Biases and toxicity encoded in their models using some of the latest responsible AI benchmarks are exposed

## Summary

In their recent work, Thibaut Lavril, Giza Cardozo, Éric Grave, and Guillaume Lample introduce the LLaMA collection of foundation language models. These models range from 7B to 65B parameters and are trained on trillions of tokens using publicly available datasets exclusively. The authors demonstrate that it is possible to train state-of-the-art models without resorting to proprietary and inaccessible datasets. They also show that a smaller model trained for longer can ultimately be cheaper at inference. The focus of this work is to train language models that achieve the best possible performance at various inference budgets by training on more tokens than what is typically used. The resulting LLaMA models outperform existing large language models (LLMs) such as GPT-3 on most benchmarks despite being smaller in size. For instance, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, while LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. The authors release all their models to the research community. Unlike other existing LLMs such as Chinchilla or PaLM which rely on data that is either not publicly available or undocumented, the authors only use publicly available data sources for training their models. This makes their work compatible with open-sourcing and democratizes access to and study of LLMs. In addition to presenting an overview of the modifications made to the transformer architecture (Vaswani et al., 2017) and their training method, the authors report the performance of their models compared with other LLMs on a set of standard benchmarks. They also expose some biases and toxicity encoded in their models using some of the latest responsible AI benchmarks. Overall, this work demonstrates that it is possible to train highly performant language models using publicly available datasets and provides a valuable resource for researchers and practitioners in the field.