# Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models

**Authors:** Steve Yadlowsky, Lyric Doshi, Nilesh Tripuraneni  
**arXiv:** [2311.00871](https://arxiv.org/abs/2311.00871)  
**Date:** 2023-11-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.00871v1/

## Key points

- Transformer models, specifically large language models (LLMs), are investigated for their in-context learning (ICL) capabilities.
- The study focuses on how well transformers can identify and learn new tasks within and outside their pretraining distribution.
- Transformers show near-optimal unsupervised model selection abilities when task families are well-represented in their pretraining data.
- However, transformers exhibit failure modes when presented with out-of-domain tasks or functions, leading to degradation of generalization abilities.
- The research suggests that the coverage of pretraining data mixtures is crucial for the impressive ICL abilities of sequence models.
- The composition and diversity of pretraining data mixtures should be considered when using transformer models for in-context learning.

## Summary

In their paper titled "Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models," authors Steve Yadlowsky, Lyric Doshi, and Nilesh Tripuraneni explore the capabilities of transformer models, particularly large language models (LLMs), in performing in-context learning (ICL). They investigate how effectively transformers can identify and learn new tasks within and outside their pretraining distribution. The authors conduct their study in a controlled setting by training transformer models on sequences of $(x, f(x))$ pairs instead of natural language. They find that transformers demonstrate near-optimal unsupervised model selection capabilities when it comes to identifying different task families and learning within them if these task families are well-represented in their pretraining data. However, the authors also observe various failure modes of transformers when presented with tasks or functions that are out-of-domain of their pretraining data. Even simple extrapolation tasks lead to degradation of the transformers' generalization abilities. This suggests that the impressive ICL abilities of high-capacity sequence models may be more closely tied to the coverage of their pretraining data mixtures rather than inherent inductive biases that enable fundamental generalization capabilities. Overall, this research highlights the importance of considering the composition and diversity of pretraining data mixtures when utilizing transformer models for in-context learning. The findings shed light on both the strengths and limitations of these models, providing valuable insights into further advancements in this field.