RULER: What's the Real Context Size of Your Long-Context Language Models?

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Another approach to the problem Cited by this paper Not relevant

   BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

   Yuri Kuratov, Aydar Bulatov, Petr Anokhin et al. · 2024 · Computation and Language · cited 304 times · Not summarized yet

   BABILong tests reasoning over facts scattered through long documents, using twenty tasks such as deduction and counting rather than RULER’s configurable synthetic retrieval, tracing, and aggregation tasks.

   Abstract

   In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts. To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. These tasks are challenging on their own, and even more demanding when the required facts are scattered across long natural text. Our evaluations sh…

2. Very close Builds on this paper Not relevant

   One ruler to measure them all: Benchmarking multilingual long-context language models

   Yekyung Kim, Jenna Russell, Marzena Karpinska et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 25 times · Not summarized yet

   ONERULER adapts RULER’s synthetic retrieval and aggregation tasks for evaluation across 26 languages, including variants with nonexistent needles.

   Abstract

   We present ONERULER, a multilingual benchmark designed to evaluate long-context language models across 26 languages. ONERULER adapts the English-only RULER benchmark (Hsieh et al., 2024) by including seven synthetic tasks that test both retrieval and aggregation, including new variations of the"needle-in-a-haystack"task that allow for the possibility of a nonexistent needle. We create ONERULER through a two-step process, first writing English instructions for each task and then collaborating with native speakers to translate them into 25 additional languages. Experiments with both open-weight and closed LLMs reveal a widening performance gap between low- and high-resource languages as contex…

3. Very close Broader context Not relevant

   NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models

   Hyeonseok Moon, Heu-Jeoung Lim · 2025 · Annual Meeting of the Association for Computational Linguistics · Not summarized yet

   NeedleChain tests whether models integrate query-relevant context without distractors, addressing a different aspect of context comprehension than RULER’s synthetic long-context tasks.

   Abstract

   Recent reports suggest that LLMs can handle increasingly long contexts. However, many existing benchmarks for context understanding embed substantial query-irrelevant content, which shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information. Under this setting, we view that current benchmarks can overestimate true context-understanding ability of LLMs. In particular, we demonstrate that when the context consists entirely of query-relevant text, even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens. To evaluate this capability more rigorously, we introduce NeedleChain, a benchmark designed to test whether…

4. Very close Another approach to the problem Cited by this paper A method it uses Not relevant

   LooGLE: Can Long-Context Language Models Understand Long Contexts?

   Jia-Qi Li, Mengmeng Wang, Zi-Long Zheng et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cited 301 times · Not summarized yet

   LooGLE benchmarks long-context understanding with long-dependency tasks and updated documents, whereas RULER uses configurable synthetic tasks to assess retrieval, tracing, and aggregation.

   Abstract

   Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation has spurred significant research efforts to enhance LLMs' long-context understanding with high-quality long-sequence benchmarks. However, prior datasets in this regard suffer from shortcomings, such as short context length compared to the context window of modern LLMs; outdated documents that have data leakage problems; and an emphasis on short dependency tasks rather than long dependency tasks. In this paper, we present LooGLE, a Long Context Generic Language Evaluation benchmark for LLMs' long context understandi…

5. Very close Another approach to the problem Cited by this paper Not relevant

   $\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens

   Xinrong Zhang, Yingfa Chen, Shengding Hu et al. · 2024 · 2023.12.15ARR · cited 454 times · Not summarized yet

   ∞Bench evaluates synthetic and realistic tasks at an average length exceeding 100K tokens, while RULER focuses on configurable synthetic tasks and model performance as context grows.

   Abstract

   Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despite recent strides in making LLMs process contexts with more than 100K tokens, there is currently a lack of a standardized benchmark to evaluate this long-context capability. Existing public benchmarks typically focus on contexts around 10K tokens, limiting the assessment and comparison of LLMs in processing longer contexts. In this paper, we propose $\infty$Bench, the first LLM benchmark featuring an average data length surpassing 100K tokens. $\infty$Bench comprises synthetic and realistic tasks spanning diverse d…

6. Very close Another approach to the problem Cited by this paper A method it uses Not relevant

   LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K

   Tao Yuan, Xue-Fei Ning, Dong Zhou et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 96 times · Not summarized yet

   LV-Eval tests single- and multi-hop question answering across five lengths up to 256K words, whereas RULER evaluates configurable synthetic retrieval, tracing, and aggregation tasks.

   Abstract

   State-of-the-art large language models (LLMs) are now claiming remarkable supported context lengths of 256k or even more. In contrast, the average context lengths of mainstream benchmarks are insufficient (5k-21k), and they suffer from potential knowledge leakage and inaccurate metrics, resulting in biased evaluation. This paper introduces LV-Eval, a challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up to 256k words. LV-Eval features two main tasks, single-hop QA and multi-hop QA, comprising 11 bilingual datasets. The design of LV-Eval has incorporated three key techniques, namely confusing facts insertion, keyword and phrase replacement, and…

7. Very close Same question Not relevant

   Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts

   Yi-Fei Yu, Qian-Wen Zhang, Ling-Feng Qiao et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 18 times · Not summarized yet

   Sequential-NIAH tests retrieval of ordered needles from long contexts, extending the needle-in-a-haystack evaluation that RULER expands beyond simple retrieval.

   Abstract

   Evaluating the ability of large language models (LLMs) to process lengthy contexts is critical, especially for retrieving query-relevant information embedded within them. We introduce Sequential-NIAH, a benchmark specifically designed to evaluate the capability of LLMs to extract sequential information items (known as \emph{needles}) from long contexts. The benchmark includes three needle generation pipelines: synthetic-temporal, real-temporal, and real-logical orders, with context lengths ranging from 8K to 128K, which comprises 14,000 samples (2,000 for testing). To facilitate the evaluation of this benchmark, we trained an evaluation model that assesses the correctness of LLM responses by…

8. Very close Same method, another problem Not relevant

   How to Train Long-Context Language Models (Effectively)

   Tianyu Gao, Alexander Wettig, Howard Yen et al. · 2024 · Computation and Language · cited 181 times · AI summary available

   How to Train Long-Context Language Models uses broad long-context evaluations, including alternatives to simple NIAH, to guide continued training and supervised fine-tuning.

   Abstract

   We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. We first establish a reliable evaluation protocol to guide model development -- Instead of perplexity or simple needle-in-a-haystack (NIAH) tests, we use a broad set of long-context tasks, and we evaluate models after SFT with instruction data as this better reveals long-context abilities. Supported by our robust evaluations, we run thorough experiments to decide the data mix for continued pre-training, the instruction tuning dataset, and many other design choices. We find that (1) code repositories and books are excellent sources of long data, but it is cr…

Prior work it builds on

1. Very close Groundwork for this paper A method it uses Not relevant

   Lost in the Middle: How Language Models Use Long Contexts

   Nelson F. Liu, Kevin Lin, John Hewitt et al. · 2023 · Computation and Language · cited 5,800 times · AI summary available

   Lost in the Middle measures how context length and information position affect retrieval and question answering, providing earlier evidence of long-context performance limits.

   Abstract

   While recent language models have the ability to take long contexts as input, relatively little is known about how well the language models use longer context. We analyze language model performance on two tasks that require identifying relevant information within their input contexts: multi-document question answering and key-value retrieval. We find that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts. Furthermore, performance substantially decreases as the input context grows longer, even for explicitly long-context models. Our a…

2. Very close Another approach to the problem A method it uses Not relevant

   L-Eval: Instituting Standardized Evaluation for Long Context Language Models

   Chen An, Shansan Gong, Ming Zhong et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cited 317 times · Not summarized yet

   L-Eval evaluates long-context models with 20 subtasks and long documents, whereas RULER uses configurable synthetic tasks to probe retrieval and reasoning behaviors.

   Abstract

   Recently, there has been growing interest in extending the context length of large language models (LLMs), aiming to effectively process long inputs of one turn or conversations with more extensive histories. While proprietary models such as GPT-4 and Claude can largely preserve the reasoning ability in an extended context, open-source models are still progressing through the early stages of development. To bridge this gap, we propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, an…

3. Very close Same method, another problem Not relevant

   Effective Long-Context Scaling of Foundation Models

   Wenhan Xiong, Jingyu Liu, Igor Molybog et al. · 2023 · Computation and Language · cited 367 times · AI summary available

   Effective Long-Context Scaling evaluates synthetic context-probing tasks alongside research benchmarks to assess models trained for longer sequences.

   Abstract

   We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5…

4. Close Another approach to the problem A method it uses Not relevant

   BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models

   Zican Dong, Tianyi Tang, Junyi Li et al. · 2023 · International Conference on Language Resources and Evaluation · cited 62 times · Not summarized yet

   BAMBOO evaluates long-text abilities across five task types and multiple length levels, rather than RULER’s configurable synthetic retrieval, tracing, and aggregation tasks.

   Abstract

   Large language models (LLMs) have achieved dramatic proficiency over NLP tasks with normal length. Recently, multiple studies have committed to extending the context length and enhancing the long text modeling capabilities of LLMs. To comprehensively evaluate the long context ability of LLMs, we propose BAMBOO, a multi-task long context benchmark. BAMBOO has been designed with four principles: comprehensive capacity evaluation, avoidance of data contamination, accurate automatic evaluation, and different length levels. It consists of 10 datasets from 5 different long text understanding tasks, i.e., question answering, hallucination detection, text sorting, language modeling, and code complet…

5. Very close Another approach to the problem Not relevant

   LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

   Yushi Bai, Xin Lv, Jia-Jie Zhang et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cited 1,880 times · Not summarized yet

   LongBench benchmarks long-context understanding across bilingual multitask datasets, while RULER uses configurable synthetic tasks to isolate specific context behaviors.

   Abstract

   Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs' long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprise…

Papers that cite it

1. Close Builds on this paper Uses this paper in depth Not relevant

   Trained Agentic Context Management

   Bryce Sandlund · 2026 · Not summarized yet

   Trained Agentic Context Management uses RULER among its evaluation references while testing a tool-using harness for long-context tasks.

   Abstract

   We study long context language models. Instead of training long context natively, or designing a long context harness, we train a model over the simplest possible harness: a tool to call itself with any specified prompt and a tool to read tokens in a range from the input context. We finetune Qwen3.6-35B-A3B on a diverse synthetic dataset using this harness. With only 8,000 tokens of context, our small model is as strong as GPT-5.4 with 1M tokens of context on the OOLONG-synth benchmark when document length exceeds 40K tokens.

2. Very close Builds on this paper Not relevant

   With a Hop, Skip, and a Prefill: How Benchmark Volatility Distorts the Accuracy of Long-Context Benchmarks and How To Combat It

   Valentijn Dymphnus van de Beek, Takeshi Yoshimura · 2026 · Workshop on Machine Learning and Systems (EuroMLSys) · Not summarized yet

   With a Hop, Skip, and a Prefill analyzes long-context benchmark volatility and cites RULER among the benchmarks it examines.

   Abstract

   Contemporary large language models now support context windows of up to millions of tokens, and this capability enables higher accuracy, new tasks, and longer conversational history. Researchers rely on long-context inference benchmarks to evaluate specific model behaviours, but practitioners still find it difficult to translate benchmark results into AI system design decisions, such as model selection and configuration for target workloads. In this work, we analyse 16 long-context benchmarks to characterise their composition in terms of tasks, prompt token sizes, and variation between prompts. We find substantial differences in context-prompt length both across and within benchmarks: the co…

3. Close Builds on this paper Not relevant

   LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

   Quang Hieu Pham, T. Nguyen, Jocelyn Qiaochu Chen et al. · 2026 · Not summarized yet

   LongHarness Bench cites RULER while introducing evaluations of harnesses that use retrieval strategies and adaptive reasoning over long contexts.

   Abstract

   Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a chal…

4. Close Builds on this paper Uses this paper in depth Not relevant

   Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size

   M. Ryu, Xin-Yu Piao, Jong-Kook Kim · 2026 · Not summarized yet

   Residual Vector-based Reconstruction cites RULER while proposing a trained-free method for recalling facts from long contexts with near-constant GPU memory.

   Abstract

   Large language models (LLMs) process long contexts, including long documents and lengthy conversations, but face token-level memory usage that increases proportionally to input length. Although model optimization and lossy prompt compression are widely used, these methods still fail to solve the long-context recall problem beyond pretrained and size-constrained context windows. This paper proposes a long-context recall method that maintains near-constant GPU memory usage as context length increases, without additional training. The main idea is to reconstruct facts using parameter activations in the LLM's feed-forward layers, which store residual vectors representing facts from the source do…

5. Close Builds on this paper Uses this paper in depth Not relevant

   Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law

   Zi-Qing Qiao, Xiao-Yue Xu, Si Zhang et al. · 2026 · Not summarized yet

   Don't Stop Long-Context Continual Pre-Training Too Early cites RULER while studying how continued-pretraining token budgets affect long-context performance.

   Abstract

   Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear. We investigate this question through modeling, monitoring, and validating the benefits of scaling LCCP. First, we establish an empirical additive power law relating long-context performance to model size and the respective token budgets of short-context pre-training and LCCP. Experiments across five model scales and more than one hundred short- and long-context budget configurations show that the fitted law reproduces the observed trends and extrapolates well to held-out model scales. Second, we introduce \textsc{Con…