RULER: What's the Real Context Size of Your Long-Context Language Models?

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Cité par cet article Pas pertinent

   BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

   Yuri Kuratov, Aydar Bulatov, Petr Anokhin et al. · 2024 · Computation and Language · cité 304 fois · Pas encore résumé

   Résumé scientifique

   In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts. To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. These tasks are challenging on their own, and even more demanding when the required facts are scattered across long natural text. Our evaluations sh…

2. Très proche Pas pertinent

   One ruler to measure them all: Benchmarking multilingual long-context language models

   Yekyung Kim, Jenna Russell, Marzena Karpinska et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 25 fois · Pas encore résumé

   Résumé scientifique

   We present ONERULER, a multilingual benchmark designed to evaluate long-context language models across 26 languages. ONERULER adapts the English-only RULER benchmark (Hsieh et al., 2024) by including seven synthetic tasks that test both retrieval and aggregation, including new variations of the"needle-in-a-haystack"task that allow for the possibility of a nonexistent needle. We create ONERULER through a two-step process, first writing English instructions for each task and then collaborating with native speakers to translate them into 25 additional languages. Experiments with both open-weight and closed LLMs reveal a widening performance gap between low- and high-resource languages as contex…

3. Très proche Pas pertinent

   NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models

   Hyeonseok Moon, Heu-Jeoung Lim · 2025 · Annual Meeting of the Association for Computational Linguistics · Pas encore résumé

   Résumé scientifique

   Recent reports suggest that LLMs can handle increasingly long contexts. However, many existing benchmarks for context understanding embed substantial query-irrelevant content, which shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information. Under this setting, we view that current benchmarks can overestimate true context-understanding ability of LLMs. In particular, we demonstrate that when the context consists entirely of query-relevant text, even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens. To evaluate this capability more rigorously, we introduce NeedleChain, a benchmark designed to test whether…

4. Très proche Cité par cet article Une méthode qu’il utilise Pas pertinent

   LooGLE: Can Long-Context Language Models Understand Long Contexts?

   Jia-Qi Li, Mengmeng Wang, Zi-Long Zheng et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cité 301 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation has spurred significant research efforts to enhance LLMs' long-context understanding with high-quality long-sequence benchmarks. However, prior datasets in this regard suffer from shortcomings, such as short context length compared to the context window of modern LLMs; outdated documents that have data leakage problems; and an emphasis on short dependency tasks rather than long dependency tasks. In this paper, we present LooGLE, a Long Context Generic Language Evaluation benchmark for LLMs' long context understandi…

5. Très proche Cité par cet article Pas pertinent

   $\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens

   Xinrong Zhang, Yingfa Chen, Shengding Hu et al. · 2024 · 2023.12.15ARR · cité 454 fois · Pas encore résumé

   Résumé scientifique

   Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despite recent strides in making LLMs process contexts with more than 100K tokens, there is currently a lack of a standardized benchmark to evaluate this long-context capability. Existing public benchmarks typically focus on contexts around 10K tokens, limiting the assessment and comparison of LLMs in processing longer contexts. In this paper, we propose $\infty$Bench, the first LLM benchmark featuring an average data length surpassing 100K tokens. $\infty$Bench comprises synthetic and realistic tasks spanning diverse d…

6. Très proche Cité par cet article Une méthode qu’il utilise Pas pertinent

   LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K

   Tao Yuan, Xue-Fei Ning, Dong Zhou et al. · 2024 · arXiv.org · preprint, non relu par les pairs · cité 96 fois · Pas encore résumé

   Résumé scientifique

   State-of-the-art large language models (LLMs) are now claiming remarkable supported context lengths of 256k or even more. In contrast, the average context lengths of mainstream benchmarks are insufficient (5k-21k), and they suffer from potential knowledge leakage and inaccurate metrics, resulting in biased evaluation. This paper introduces LV-Eval, a challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up to 256k words. LV-Eval features two main tasks, single-hop QA and multi-hop QA, comprising 11 bilingual datasets. The design of LV-Eval has incorporated three key techniques, namely confusing facts insertion, keyword and phrase replacement, and…

7. Très proche Pas pertinent

   Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts

   Yi-Fei Yu, Qian-Wen Zhang, Ling-Feng Qiao et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · cité 18 fois · Pas encore résumé

   Résumé scientifique

   Evaluating the ability of large language models (LLMs) to process lengthy contexts is critical, especially for retrieving query-relevant information embedded within them. We introduce Sequential-NIAH, a benchmark specifically designed to evaluate the capability of LLMs to extract sequential information items (known as \emph{needles}) from long contexts. The benchmark includes three needle generation pipelines: synthetic-temporal, real-temporal, and real-logical orders, with context lengths ranging from 8K to 128K, which comprises 14,000 samples (2,000 for testing). To facilitate the evaluation of this benchmark, we trained an evaluation model that assesses the correctness of LLM responses by…

8. Très proche Pas pertinent

   How to Train Long-Context Language Models (Effectively)

   Tianyu Gao, Alexander Wettig, Howard Yen et al. · 2024 · Computation and Language · cité 181 fois · Résumé IA disponible

   Résumé scientifique

   We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. We first establish a reliable evaluation protocol to guide model development -- Instead of perplexity or simple needle-in-a-haystack (NIAH) tests, we use a broad set of long-context tasks, and we evaluate models after SFT with instruction data as this better reveals long-context abilities. Supported by our robust evaluations, we run thorough experiments to decide the data mix for continued pre-training, the instruction tuning dataset, and many other design choices. We find that (1) code repositories and books are excellent sources of long data, but it is cr…

Les travaux antérieurs sur lesquels il s’appuie

1. Très proche Une méthode qu’il utilise Pas pertinent

   Lost in the Middle: How Language Models Use Long Contexts

   Nelson F. Liu, Kevin Lin, John Hewitt et al. · 2023 · Computation and Language · cité 5 800 fois · Résumé IA disponible

   Résumé scientifique

   While recent language models have the ability to take long contexts as input, relatively little is known about how well the language models use longer context. We analyze language model performance on two tasks that require identifying relevant information within their input contexts: multi-document question answering and key-value retrieval. We find that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts. Furthermore, performance substantially decreases as the input context grows longer, even for explicitly long-context models. Our a…

2. Très proche Une méthode qu’il utilise Pas pertinent

   L-Eval: Instituting Standardized Evaluation for Long Context Language Models

   Chen An, Shansan Gong, Ming Zhong et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cité 317 fois · Pas encore résumé

   Résumé scientifique

   Recently, there has been growing interest in extending the context length of large language models (LLMs), aiming to effectively process long inputs of one turn or conversations with more extensive histories. While proprietary models such as GPT-4 and Claude can largely preserve the reasoning ability in an extended context, open-source models are still progressing through the early stages of development. To bridge this gap, we propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, an…

3. Très proche Pas pertinent

   Effective Long-Context Scaling of Foundation Models

   Wenhan Xiong, Jingyu Liu, Igor Molybog et al. · 2023 · Computation and Language · cité 367 fois · Résumé IA disponible

   Résumé scientifique

   We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5…

4. Proche Une méthode qu’il utilise Pas pertinent

   BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models

   Zican Dong, Tianyi Tang, Junyi Li et al. · 2023 · International Conference on Language Resources and Evaluation · cité 62 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) have achieved dramatic proficiency over NLP tasks with normal length. Recently, multiple studies have committed to extending the context length and enhancing the long text modeling capabilities of LLMs. To comprehensively evaluate the long context ability of LLMs, we propose BAMBOO, a multi-task long context benchmark. BAMBOO has been designed with four principles: comprehensive capacity evaluation, avoidance of data contamination, accurate automatic evaluation, and different length levels. It consists of 10 datasets from 5 different long text understanding tasks, i.e., question answering, hallucination detection, text sorting, language modeling, and code complet…

5. Très proche Pas pertinent

   LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

   Yushi Bai, Xin Lv, Jia-Jie Zhang et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cité 1 880 fois · Pas encore résumé

   Résumé scientifique

   Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs' long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprise…

Les articles qui le citent

1. Proche Utilise cet article en profondeur Pas pertinent

   Trained Agentic Context Management

   Bryce Sandlund · 2026 · Pas encore résumé

   Résumé scientifique

   We study long context language models. Instead of training long context natively, or designing a long context harness, we train a model over the simplest possible harness: a tool to call itself with any specified prompt and a tool to read tokens in a range from the input context. We finetune Qwen3.6-35B-A3B on a diverse synthetic dataset using this harness. With only 8,000 tokens of context, our small model is as strong as GPT-5.4 with 1M tokens of context on the OOLONG-synth benchmark when document length exceeds 40K tokens.

2. Très proche Pas pertinent

   With a Hop, Skip, and a Prefill: How Benchmark Volatility Distorts the Accuracy of Long-Context Benchmarks and How To Combat It

   Valentijn Dymphnus van de Beek, Takeshi Yoshimura · 2026 · Workshop on Machine Learning and Systems (EuroMLSys) · Pas encore résumé

   Résumé scientifique

   Contemporary large language models now support context windows of up to millions of tokens, and this capability enables higher accuracy, new tasks, and longer conversational history. Researchers rely on long-context inference benchmarks to evaluate specific model behaviours, but practitioners still find it difficult to translate benchmark results into AI system design decisions, such as model selection and configuration for target workloads. In this work, we analyse 16 long-context benchmarks to characterise their composition in terms of tasks, prompt token sizes, and variation between prompts. We find substantial differences in context-prompt length both across and within benchmarks: the co…

3. Proche Pas pertinent

   LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

   Quang Hieu Pham, T. Nguyen, Jocelyn Qiaochu Chen et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a chal…

4. Proche Utilise cet article en profondeur Pas pertinent

   Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size

   M. Ryu, Xin-Yu Piao, Jong-Kook Kim · 2026 · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) process long contexts, including long documents and lengthy conversations, but face token-level memory usage that increases proportionally to input length. Although model optimization and lossy prompt compression are widely used, these methods still fail to solve the long-context recall problem beyond pretrained and size-constrained context windows. This paper proposes a long-context recall method that maintains near-constant GPU memory usage as context length increases, without additional training. The main idea is to reconstruct facts using parameter activations in the LLM's feed-forward layers, which store residual vectors representing facts from the source do…

5. Proche Utilise cet article en profondeur Pas pertinent

   Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law

   Zi-Qing Qiao, Xiao-Yue Xu, Si Zhang et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear. We investigate this question through modeling, monitoring, and validating the benefits of scaling LCCP. First, we establish an empirical additive power law relating long-context performance to model size and the respective token budgets of short-context pre-training and LCCP. Experiments across five model scales and more than one hundred short- and long-context budget configurations show that the fitted law reproduces the observed trends and extrapolates well to held-out model scales. Second, we introduce \textsc{Con…