# Parallel Context Windows Improve In-Context Learning of Large Language Models

**Authors:** Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham  
**arXiv:** [2212.10947](https://arxiv.org/abs/2212.10947)  
**Date:** 2022-12-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.10947v1/

## Key points

- Large Language Models (LLMs) have limitations in processing large amounts of text at inference time due to restricted context windows
- Existing solutions involve training specialized architectures, but they are often smaller in size and have memory constraints
- The authors propose a method called Parallel Context Windows (PCW) to overcome the context window restriction without additional training
- PCW involves dividing a long context into chunks or "windows," restricting attention mechanisms within each window, and reusing positional embeddings across windows
- PCW improves in-context learning with LLM models ranging from 750 million to 178 billion parameters
- PCW shows promise in enhancing in-context learning for various applications with diverse input and output spaces
- PCW can be applied to any off-the-shelf LLM without extensive modifications or specialized architectures
- Figure 1 illustrates how PCW improves in-context learning accuracy compared to single context window approaches using the BANKING-77 intent classification dataset
- Figure 2 provides an illustration of how PCW exposes an LLM to multiple context windows during generation
- PCW offers a solution that can be applied to any off-the-shelf LLM, enabling them to process longer text sequences without specialized training

## Summary

The paper discusses the limitations of Large Language Models (LLMs) in processing large amounts of text at inference time due to their restricted context windows. Existing solutions involve training specialized architectures, but they are often smaller in size and have memory constraints. To address this issue, the authors propose a method called Parallel Context Windows (PCW), which allows any off-the-shelf LLM to overcome the context window restriction without additional training. PCW involves dividing a long context into chunks or "windows" that fit within the architecture, restricting attention mechanisms within each window, and reusing positional embeddings across windows. The authors test the effectiveness of PCW on in-context learning with LLM models ranging from 750 million to 178 billion parameters. They observe significant improvements in tasks with diverse input and output spaces, suggesting that PCW can enhance in-context learning for various applications. The results motivate further exploration of PCW as a method for applying off-the-shelf LLMs in settings that require processing long text sequences. In conclusion, the paper introduces PCW as a simple yet effective approach for expanding the scope of text accessible by LLMs during inference. The authors highlight two promising directions for future work: investigating the application of PCW in other settings requiring mainstream LLMs over long text sequences and exploring the potential benefits of further training an LLM with parallel context windows. Overall, PCW shows promise in improving the capabilities of off-the-shelf LLMs without extensive modifications or specialized architectures. Figure 1 illustrates how PCW improves in-context learning accuracy compared to single context window approaches using the BANKING-77 intent classification dataset. Figure 2 provides an illustration of how PCW exposes an LLM to multiple context windows during generation. The paper acknowledges previous efforts to address similar challenges through dedicated architectures but notes their limitations and lack of general applicability. The introduction of PCW offers a solution that can be applied to any off-the-shelf LLM, enabling them to process text longer than their original context window without the need for specialized training. In summary, the paper presents PCW as a method to overcome the context window limitation in LLMs, showcasing its effectiveness in improving in-context learning with potential applications across various settings warranting further investigation.