# GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

**Authors:** Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern  
**arXiv:** [2112.06905](https://arxiv.org/abs/2112.06905)  
**Date:** 2021-12-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.06905v1/

## Key points

- GLaM (Generalist Language Model) is a family of language models that uses a sparsely activated mixture-of-experts architecture.
- GLaM models have significantly reduced training costs compared to dense variants.
- The largest GLaM model has 1.2 trillion parameters, 7 times larger than GPT-3, but consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference.
- A high-quality dataset of 1.6 trillion tokens representative of various natural language use cases is constructed to train the GLaM models.
- The dataset consists mainly of web pages, ranging from professional writing to low-quality comment and forum pages.
- A text quality classifier is developed to filter out low-quality webpages and create a high-quality web corpus.
- Ethical considerations related to large language models are addressed in the study, including potential biases associated with filtering marginalized groups' text and reinforcing unfair bias when automatically filtering low-quality content from web text collections.
- Existing charters from organizations like OpenAI, Google, Facebook, and Microsoft are highlighted as addressing these ethical concerns.
- The proposed GLaM models demonstrate efficient scaling capabilities while considering ethical considerations associated with training data quality and task applications.

## Summary

In this paper, the authors propose and develop a family of language models called GLaM (Generalist Language Model) that utilizes a sparsely activated mixture-of-experts architecture. This allows for scaling the model capacity while significantly reducing training costs compared to dense variants. The largest GLaM model has 1.2 trillion parameters, approximately 7 times larger than GPT-3, but consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference. To train the GLaM models, a high-quality dataset of 1.6 trillion tokens representative of various natural language use cases is constructed. The dataset consists mainly of web pages, ranging from professional writing to low-quality comment and forum pages. A text quality classifier is developed to filter out low-quality webpages and create a high-quality web corpus. This classifier is trained to distinguish between curated text (such as Wikipedia, books, and selected websites) and other webpages. The filtered subset of webpages is combined with books, Wikipedia pages, public domain social media conversations, and other data sources to create the final GLaM dataset. The importance of data filtering in improving model quality is analyzed in the study. Ethical considerations related to large language models are also addressed in this work. The authors emphasize the significance of high-quality pre-training corpora for achieving good model performance. They acknowledge the potential for over-filtering text associated with marginalized groups and reinforcing unfair bias when automatically filtering low-quality content from web text collections. The authors advocate for a more thoughtful approach when deciding which tasks should be pursued using language models and which tasks should be avoided due to ethical implications. They highlight existing charters from organizations like OpenAI, Google, Facebook, and Microsoft that address these concerns. Overall, the proposed GLaM models demonstrate efficient scaling capabilities while considering ethical considerations associated with training data quality and task applications.