# RULER: What's the Real Context Size of Your Long-Context Language Models?

**Authors:** Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Boris Ginsburg  
**arXiv:** [2404.06654](https://arxiv.org/abs/2404.06654)  
**Date:** 2024-04-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.06654v1/

## Key points

- The NIAH test is commonly used to evaluate long-context language models by assessing their ability to retrieve specific information from lengthy distractor texts.
- RULER is introduced as a synthetic benchmark that expands upon the NIAH test, offering customizable sequence lengths and task complexities for a more comprehensive evaluation framework.
- RULER includes diverse types and quantities of needles, multi-hop tracing, aggregation tasks, and other categories to test behaviors beyond simple context-based retrieval.
- Evaluation of ten long-context LMs across 13 tasks in RULER showed significant performance drops as context length increased, with only four models maintaining satisfactory performance at larger context sizes.
- RULER is open source and aims to encourage comprehensive evaluation of long-context LMs by providing a flexible and controllable evaluation framework across various task categories like retrieval, multi-hop tracing, aggregation, and question answering.

## Summary

In the realm of evaluating long-context language models (LMs), the needle-in-a-haystack (NIAH) test has been a popular choice. This test assesses the ability of LMs to retrieve specific information ("needles") from lengthy distractor texts ("haystacks"). However, while effective for evaluating basic retrieval capabilities, the NIAH test only scratches the surface of long-context understanding. To address this limitation and provide a more comprehensive evaluation framework, we introduce RULER, a synthetic benchmark with customizable sequence lengths and task complexities. have become increasingly important in natural language processing tasks. In order to evaluate their performance accurately,  offers a more comprehensive approach compared to traditional methods such as the  

RULER expands upon the traditional NIAH test by incorporating diverse types and quantities of needles, as well as introducing new task categories such as multi-hop tracing and aggregation. These additions allow for testing behaviors beyond simple context-based retrieval. In our evaluation of ten long-context LMs across 13 representative tasks in RULER, we observed significant performance drops as context length increased. Despite claims of supporting context sizes of 32K tokens or more, only four models (GPT-4, Command-R, Yi-34B, and Mixtral) maintained satisfactory performance at that length. Further analysis of Yi-34B revealed room for improvement as input length and task complexity increased. We have made RULER open source to encourage comprehensive evaluation of long-context LMs. Additionally, has expanded on existing benchmarks like ZeroSCROLLS, L-Eval, BAMBOO, InfiniteBench, to offer a more flexible and controllable evaluation framework for long-context language models. Within RULER's four task categories - retrieval, multi-hop tracing, aggregation, and question answering - tasks are configurable for varying lengths and complexities. The retrieval tasks in RULER build upon the NIAH test criteria by requiring models to be adept at retrieving different types of information from various contexts while disregarding distractions. Overall,  provides a robust platform for evaluating long-context language models across a range of tasks and complexities.