# On the generalization of language models from in-context learning and finetuning: a controlled study

**Authors:** Andrew K. Lampinen, Arslan Chaudhry, Stephanie C. Y. Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, James L. McClelland  
**arXiv:** [2505.00661](https://arxiv.org/abs/2505.00661)  
**Date:** 2025-05-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.00661v1/

## Key points

- Study focused on generalization of language models from in-context learning and fine-tuning
- Constructed novel datasets to evaluate and improve models' ability to generalize
- Used reversal dataset with descriptions of fictional celebrities and a semantic structure with replaced terms
- In data-matched settings, in-context learning showed more flexible generalization than fine-tuning
- Fine-tuning could also generalize effectively within a larger structure of knowledge
- Proposed method involving adding in-context inferences to fine-tuning data to enhance generalization

## Summary

In the study on the generalization of language models from in-context learning and fine-tuning, researchers explored the differences in generalization between these two learning methods. They constructed novel datasets to evaluate and improve models' ability to generalize from fine-tuning data by isolating knowledge in the dataset from that in pretraining. The datasets were designed to create clean tests of generalization, exposing pretrained large models to controlled subsets of information either in context or through fine-tuning. One dataset used was the reversal dataset proposed by Berglund et al., containing descriptions of fictional celebrities with names either preceding or following the description. Another benchmark involved a semantic structure with a hierarchy of properties and relations based on real-world categories and relations. To make this structure novel to pretrained models, all nouns, adjectives, and verbs were replaced with nonsense terms. Despite potential tokenization challenges, short nonsense words were generated using plausible combinations of phonemes for English. For training, facts about the semantic hierarchy were assembled into synthetic articles resembling Wikipedia entries, along with QA examples to maintain question-answering capabilities during fine-tuning. The train set ensured that all necessary facts for test questions were presented at least once. Overall, the study found that in data-matched settings, in-context learning exhibited more flexible generalization than fine-tuning. However, there were also cases where fine-tuning could generalize effectively within a larger structure of knowledge. To enhance generalization from fine-tuning, a method involving adding in-context inferences to finetuning data was proposed and shown to improve performance across various datasets and benchmarks. These findings have implications for understanding the inductive biases of different learning modes in language models and offer practical insights for improving their performance.