# Lost in the Middle: How Language Models Use Long Contexts

**Authors:** Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang  
**arXiv:** [2307.03172](https://arxiv.org/abs/2307.03172)  
**Date:** 2023-07-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.03172v1/

## Key points

- Language models struggle to effectively utilize longer input contexts
- Performance is best when relevant information is at the beginning or end of the context
- Performance significantly degrades when accessing information in the middle of long contexts
- Even explicitly long-context models experience decreased performance as context length increases
- Insights into language model behavior and new evaluation protocols are provided
- Contributes to a better understanding of language model behavior and advancements in modeling longer contexts

## Summary

The paper titled "Lost in the Middle: How Language Models Use Long Contexts" by Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang explores the performance of language models on tasks that require identifying relevant information within long input contexts. Recent language models have the ability to process long contexts as input; however there is limited understanding of how effectively they utilize this longer context. The authors specifically analyze language model performance on multi-document question answering and key-value retrieval tasks. The findings reveal that language models tend to perform best when the relevant information is located at the beginning or end of the input context. However their performance significantly degrades when they need to access relevant information in the middle of long contexts. Even explicitly long-context models experience a substantial decrease in performance as the input context grows longer. This analysis provides valuable insights into how language models utilize their input context and introduces new evaluation protocols for future long-context models. By shedding light on these aspects, this research contributes to a better understanding of language model behavior and lays the foundation for further advancements in modeling longer contexts.