# Many-Shot In-Context Learning

**Authors:** Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, Hugo Larochelle  
**arXiv:** [2404.11018](https://arxiv.org/abs/2404.11018)  
**Date:** 2024-04-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.11018v1/

## Key points

- Large Language Models (LLMs) capabilities in few-shot and many-shot in-context learning (ICL)
- Expanding context windows to include hundreds or thousands of examples leads to significant performance gains
- Availability of human-generated examples can be a limiting factor for many-shot ICL
- Introduction of Reinforced and Unsupervised ICL settings as alternatives to human examples
- Reinforced ICL uses model-generated chain-of-thought rationales, while Unsupervised ICL prompts with domain-specific questions
- Both Reinforced and Unsupervised ICL are effective for complex reasoning tasks in the many-shot regime
- Many-shot learning overcomes pretraining biases and learns high-dimensional functions with numerical inputs
- Next-token prediction loss may not always indicate downstream ICL performance reliably
- Impact of scaling examples on abstractive summarization tasks using XSum dataset, showing improved performance up to 50 shots before deterioration
- Comparison with models fine-tuned for summarization like PEGASUS and mT5, which show continuous improvement with more shots from XSum
- Evaluation of commonsense planning abilities of LLMs in Logistics domain, showing promise in generating simple plans within cities using trucks and airplanes through many-shot ICL
- Training LLMs to learn code verifiers in-context through reward modeling enhances reasoning abilities, indicating potential improvements in commonsense planning abilities
- Many-shot learning enhances LLM performance across various tasks and domains, advancing natural language understanding and reasoning capabilities

## Summary

In this study, we explore the capabilities of Large Language Models (LLMs) in few-shot and many-shot in-context learning (ICL). We find that expanding context windows to include hundreds or thousands of examples in the many-shot regime leads to significant performance gains across various generative and discriminative tasks. However, the availability of human-generated examples can be a limiting factor for many-shot ICL. To address this limitation, we introduce two new settings: Reinforced and Unsupervised ICL. Reinforced ICL utilizes model-generated chain-of-thought rationales instead of human examples, while Unsupervised ICL prompts the model with domain-specific questions without any rationales. Our experiments show that both Reinforced and Unsupervised ICL are effective in the many-shot regime, particularly for complex reasoning tasks. Additionally, we demonstrate that many-shot learning is capable of overcoming pretraining biases and learning high-dimensional functions with numerical inputs. Interestingly, our analysis reveals that next-token prediction loss may not always be a reliable indicator of downstream ICL performance. Furthermore, we investigate the impact of scaling examples for ICL on abstractive summarization tasks using the XSum dataset. By increasing the number of in-context examples up to 50 shots, we observe improved performance before seeing a deterioration. In contrast, models fine-tuned for summarization such as PEGASUS and mT5 typically show continuous improvement with more shots from XSum. We also delve into commonsense planning abilities of LLMs by evaluating their performance on planning problems in the Logistics domain. Many-shot ICL shows promise in improving their ability to generate simple plans within cities using trucks and airplanes. Lastly, we explore reward modeling by training LLMs to learn code verifiers in-context. This approach aims to enhance reasoning abilities through test-time verification processes. Our results indicate potential improvements in commonsense planning abilities through many-shot ICL. Overall, our study highlights the effectiveness of many-shot learning in enhancing LLM performance across various tasks and domains, showcasing its potential for advancing natural language understanding and reasoning capabilities.