# Are Emergent Abilities of Large Language Models a Mirage?

**Authors:** Rylan Schaeffer, Brando Miranda, Sanmi Koyejo  
**arXiv:** [2304.15004](https://arxiv.org/abs/2304.15004)  
**Date:** 2023-04-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.15004v1/

## Key points

- Growing interest in emergent abilities of large language models
- Emergent abilities defined as skills not present in smaller-scale models but appear suddenly and unpredictably when models reach a certain size
- New study challenges notion that emergent abilities are fundamental properties of scaling AI models
- Researchers propose alternative explanation: for a particular task and model family, one can choose a metric that leads to the inference of an emergent ability or another metric that does not
- Three complementary analyses conducted using various deep network architectures such as convolutional, autoencoder, and transformers
- Results showed that different metrics could lead to vastly different conclusions about whether an emergent ability exists or not
- Study challenges previous claims about large language models' capabilities
- Highlights importance of carefully selecting metrics when analyzing model outputs
- Raises questions about how researchers should interpret results from previous studies claiming to have found emergent abilities in large language models.

## Summary

In recent years, there has been a growing interest in the emergent abilities of large language models. These abilities are defined as skills that are not present in smaller-scale models but appear suddenly and unpredictably when models reach a certain size. However, a new study challenges this notion and suggests that these abilities may not be fundamental properties of scaling AI models. Instead, the researchers propose an alternative explanation: for a particular task and model family, one can choose a metric that leads to the inference of an emergent ability or another metric that does not. To test their hypothesis, the researchers conducted three complementary analyses using various deep network architectures such as convolutional, autoencoder, and transformers. In the first analysis, they made three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities. The results confirmed their predictions and showed that different metrics could lead to vastly different conclusions about whether an emergent ability exists or not. In the second analysis, they conducted a meta-analysis of emergent abilities on BIG-Bench and made two predictions about metric choices. Again, their findings supported their hypothesis that different metrics could lead to different interpretations of model behavior. Finally, in the third analysis, they showed how similar metric decisions suggest apparent emergent abilities on vision tasks in diverse deep network architectures. Overall, all three analyses provided strong supporting evidence for their alternative explanation. The implications of this study are significant because it challenges previous claims about large language models' capabilities and highlights the importance of carefully selecting metrics when analyzing model outputs. It also raises questions about how researchers should interpret results from previous studies claiming to have found emergent abilities in large language models.