LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Same question Not relevant

   Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability

   Tsz-Ting Chung, Leyang Cui, Le-Mao Liu et al. · 2024 · Conference on Empirical Methods in Natural Language Processing · cited 12 times · Not summarized yet

   Selection-p studies task-agnostic prompt compression, using self-supervised pretraining to improve faithfulness and transferability without external training data.

   Abstract

   Large Language Models (LLMs) have demonstrated impressive capabilities in a wide range of natural language processing tasks when leveraging in-context learning. To mitigate the additional computational and financial costs associated with in-context learning, several prompt compression methods have been proposed to compress the in-context learning prompts. Despite their success, these methods face challenges with transferability due to model-specific compression, or rely on external training data, such as GPT-4. In this paper, we investigate the ability of LLMs to develop a unified compression method that discretizes uninformative tokens, utilizing a self-supervised pre-training technique. By…

2. Very close Another approach to the problem Not relevant

   Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt

   Zhaozhuo Xu, Zirui Liu, Beidi Chen et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 47 times · AI summary available

   Compress, Then Prompt targets transferable prompt compression to balance LLM inference accuracy and efficiency.

   Abstract

   Large Language Models (LLMs), armed with billions of parameters, exhibit exceptional performance across a wide range of Natural Language Processing (NLP) tasks. However, they present a significant computational challenge during inference, especially when deploying on common hardware such as single GPUs. As such, minimizing the latency of LLM inference by curtailing computational and memory requirements, though achieved through compression, becomes critically important. However, this process inevitably instigates a trade-off between efficiency and accuracy, as compressed LLMs typically experience a reduction in predictive precision. In this research, we introduce an innovative perspective: to…

3. Very close Same question Cites this paper Uses this paper in depth Not relevant

   Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor

   Ivan Zakazov, Alexander Sharipov, Berke Argin et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Cmprsr evaluates and improves token-level, question-agnostic prompt compression, emphasizing preservation of important information and adherence to a compression rate.

   Abstract

   Motivated by the high costs of using black-box Large Language Models (LLMs), we introduce a novel prompt compression paradigm, under which we use smaller LLMs to compress inputs for the larger ones. We present the first comprehensive LLM-as-a-compressor benchmark spanning 25 open- and closed-source models, which reveals significant disparity in models'compression ability in terms of (i) preserving semantically important information (ii) following the user-provided compression rate (CR). We further improve the performance of gpt-4.1-mini, the best overall vanilla compressor, with Textgrad-based compression meta-prompt optimization. We also identify the most promising open-source vanilla LLM -…

4. Very close Another approach to the problem Cited by this paper Used in depth by this paper A method it uses Not relevant

   LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

   Huiqiang Jiang, Qianhui Wu, Xufang Luo et al. · 2023 · Computation and Language · cited 658 times · AI summary available

   LongLLMLingua compresses prompts for long-context tasks by considering key-information density and position to improve efficiency and model performance.

   Abstract

   In long context scenarios, large language models (LLMs) face three main challenges: higher computational/financial cost, longer latency, and inferior performance. Some studies reveal that the performance of LLMs depends on both the density and the position of the key information (question relevant) in the input prompt. Inspired by these findings, we propose LongLLMLingua for prompt compression towards improving LLMs' perception of the key information to simultaneously address the three challenges. We conduct evaluation on a wide range of long context scenarios including single-/multi-document QA, few-shot learning, summarization, synthetic tasks, and code completion. The experimental results…

5. Very close Builds on this paper Cites this paper Not relevant

   Prompt Compression for Diffusion Large Language Models: An Empirical Evaluation of LLMLingua-2 and LLaDA

   Sterling Huang, Abigayle Brown, Jiyoo Noh et al. · 2026 · International Conference on Advances in Information Technology · cited 1 time · Not summarized yet

   Prompt Compression for Diffusion Large Language Models tests whether LLMLingua-2 preserves task performance when transferred to LLaDA.

   Abstract

   Prompt compression reduces inference cost and context length in large language models, but prior evaluations focus mainly on autoregressive architectures. This study examines whether LLMLingua-2 transfers effectively to diffusion large language models (DLLMs), specifically LLaDA-8B-Instruct. We evaluate GSM8K, DUC2004, and ShareGPT using 250 prompts per dataset at an approximate 50% compression ratio, covering mathematical reasoning, prompt reconstruction, and summarization. Outputs from original, compressed, and reconstructed prompts are compared using exact-match accuracy, BLEU, ROUGE, and BERTScore. Results show that high semantic preservation does not necessarily ensure stable downstream…

6. Very close Same method, another problem Cites this paper Not relevant

   CompactPrompt: A Unified Pipeline for Prompt Data Compression in LLM Workflows

   J. Choi, Jia-Yang Zhao, Jeel Shah et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   CompactPrompt also removes low-information prompt tokens, while extending compression to attached files and numerical data in agentic workflows.

   Abstract

   Large Language Models (LLMs) deliver powerful reasoning and generation capabilities but incur substantial run-time costs when operating in agentic workflows that chain together lengthy prompts and process rich data streams. We introduce CompactPrompt, an end-to-end pipeline that merges hard prompt compression with lightweight file-level data compression. CompactPrompt first prunes low-information tokens from prompts using self-information scoring and dependency-based phrase grouping. In parallel, it applies n-gram abbreviation to recurrent textual patterns in attached documents and uniform quantization to numerical columns, yielding compact yet semantically faithful representations. Integrat…

7. Very close Same question Not relevant

   Learning to Compress Prompt in Natural Language Formats

   Yu-Neng Chuang, Tian-Wei Xing, Chia-yuan Chang et al. · 2024 · North American Chapter of the Association for Computational Linguistics · cited 66 times · Not summarized yet

   Learning to Compress Prompt in Natural Language Formats seeks transferable compression of lengthy prompts into natural-language forms.

   Abstract

   Large language models (LLMs) are great at processing multiple natural language processing tasks, but their abilities are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. Deploying LLMs with precise and informative context helps users process large-scale datasets more effectively and cost-efficiently. Existing works rely on compressing long prompt contexts into soft prompts. However, soft prompt compression encounters limitations in transferability across different LLMs, especially API-based LLMs. To this end, this work aims to compress lengthy prompts in the form of natural language with LLM transferability. This poses t…

8. Very close Review Not relevant

   An Empirical Study on Prompt Compression for Large Language Models

   Zheng Zhang, Jin-Yi Li, Yi-Huai Lan et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 11 times · Not summarized yet

   An Empirical Study compares six prompt-compression methods across datasets and analyzes response quality, hallucinations, and omitted words.

   Abstract

   Prompt engineering enables Large Language Models (LLMs) to perform a variety of tasks. However, lengthy prompts significantly increase computational complexity and economic costs. To address this issue, we study six prompt compression methods for LLMs, aiming to reduce prompt length while maintaining LLM response quality. In this paper, we present a comprehensive analysis covering aspects such as generation performance, model hallucinations, efficacy in multimodal tasks, word omission analysis, and more. We evaluate these methods across 13 datasets, including news, scientific articles, commonsense QA, math QA, long-context QA, and VQA datasets. Our experiments reveal that prompt compression …

Prior work it builds on

1. Very close Same question A method it uses Not relevant

   Discrete Prompt Compression With Reinforcement Learning

   Hoyoun Jung, Kyung-Joong Kim · 2023 · IEEE Access · cited 58 times · Not summarized yet

   Discrete Prompt Compression With Reinforcement Learning learns to edit prompts directly to reduce length while retaining their usefulness across language models.

   Abstract

   Compressed prompts aid instruction-tuned language models (LMs) in overcoming context window limitations and reducing computational costs. Existing methods, which are primarily based on training embeddings, face various challenges associated with interpretability, the fixed number of embedding tokens, reusability across different LMs, and inapplicability when interacting with black-box APIs. This study proposes prompt compression with reinforcement learning (PCRL), which is a discrete prompt compression method that addresses these issues. The proposed PCRL method utilizes a computationally efficient policy network that edits prompts directly. The training approach employed in the proposed PCR…

2. Close Another approach to the problem Not relevant

   Learning to Compress Prompts with Gist Tokens

   Jesse Mu, Xiang Lisa Li, Noah D. Goodman · 2023 · Neural Information Processing Systems · cited 534 times · Not summarized yet

   Learning to Compress Prompts with Gist Tokens compresses prompts into reusable cached tokens, rather than extracting and classifying prompt tokens.

   Abstract

   Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of"gist"tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt …

3. Close Broader context Not relevant

   Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models

   D. Wingate, M. Shoeybi, Taylor Sorensen · 2022 · Conference on Empirical Methods in Natural Language Processing · cited 128 times · Not summarized yet

   Prompt Compression and Contrastive Conditioning studies compressed prompts as a means to retain conditioning information and control generation, including toxicity reduction.

   Abstract

   We explore the idea of compressing the prompts used to condition language models, and show that compressed prompts can retain a substantive amount of information about the original prompt. For severely compressed prompts, while fine-grained information is lost, abstract information and general sentiments can be retained with surprisingly few parameters, which can be useful in the context of decode-time algorithms for controllability and toxicity reduction. We explore contrastive conditioning to steer language model generation towards desirable text and away from undesirable text, and find that some complex prompts can be effectively compressed into a single token to guide generation. We also…

4. Close Another approach to the problem Not relevant

   Adapting Language Models to Compress Contexts

   A. Chevalier, Alexander Wettig, Anirudh Ajith et al. · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 465 times · Not summarized yet

   Adapting Language Models to Compress Contexts represents long documents as learned summary vectors used as soft prompts.

   Abstract

   Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These models are capable of compressing long contexts into compact summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents are processed in segments and summary vectors from all previous segments are used in language modeling. We fine-tune OPT models on sequences of up to 30,720 tokens and show that AutoCompressors can util…

5. Related Shared data, tool or method Used in depth by this paper A method it uses Not relevant

   LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

   Yushi Bai, Xin Lv, Jia-Jie Zhang et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cited 1,888 times · Not summarized yet

   LongBench provides a bilingual, multitask benchmark for long-context understanding, and LLMLingua-2 evaluates on LongBench.

   Abstract

   Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs' long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprise…

Papers that cite it

1. Very close Same question Uses this paper in depth Not relevant

   Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference

   Cornelius Kummer, Lena Jurkschat, Michael Färber et al. · 2026 · European Conference on Information Retrieval · cited 9 times · Not summarized yet

   Prompt Compression in the Wild measures prompt-compression latency, rate adherence, and output quality to assess effects on inference speed.

   Abstract

   With the wide adoption of language models for IR -- and specifically RAG systems -- the latency of the underlying LLM becomes a crucial bottleneck, since the long contexts of retrieved passages lead large prompts and therefore, compute increase. Prompt compression, which reduces the size of input prompts while aiming to preserve performance on downstream tasks, has established itself as a cost-effective and low-latency method for accelerating inference in large language models. However, its usefulness depends on whether the additional preprocessing time during generation is offset by faster decoding. We present the first systematic, large-scale study of this trade-off, with thousands of runs…

2. Very close Another approach to the problem Uses this paper in depth Not relevant

   PC-SubMax: Efficient Prompt Compression via Regularized Submodular Maximization

   Zi-Yi Zhang, Shuang Cui, Hao-Tian Zhang et al. · 2026 · cited 1 time · Not summarized yet

   PC-SubMax selects prompt content using regularized submodular maximization to account for inter-sentence redundancy and reduce inference overhead.

   Abstract

   While large language models (LLMs) are increasingly deployed in long-context scenarios, lengthy prompts can increase inference costs and latency and exacerbate the ``lost-in-the-middle''phenomenon. Selective prompt compression offers a model-agnostic approach to alleviating these issues. However, methods based on fixed token- or sentence-level importance scores may overlook how content contributions change with the selected subset, limiting their ability to account for inter-sentence redundancy. Compression procedures that rely on autoregressive LLM scoring can also introduce substantial overhead. We propose PC-SubMax, a theoretically grounded framework that formulates selective prompt compr…

3. Very close Same question Uses this paper in depth Not relevant

   Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression

   Warren Johnson · 2026 · arXiv.org · preprint, not peer reviewed · cited 4 times · Not summarized yet

   Compression Method Matters studies how prompt compression changes output length and total inference cost across benchmarks.

   Abstract

   Prompt compression is often evaluated by input-token reduction, but its real deployment impact depends on how compression changes output length and total inference cost. We present a controlled replication and extension study of benchmark-dependent output dynamics under aggressive compression, covering 5,400 API calls across three benchmarks and multiple providers. To explain conflicting prior observations, we formalize instruction survival probability (Psi), a structural metric that captures whether task-critical prompt segments remain after truncation. Results show a strong benchmark effect: under r=0.3, DeepSeek exhibits severe output expansion on MBPP (56x, Psi approx 0.15) but substanti…

4. Very close Another approach to the problem Not relevant

   SCOPE: A Generative Approach for LLM Prompt Compression

   Tinghui Zhang, Yifan Wang, Daisy Zhe Wang · 2025 · arXiv.org · preprint, not peer reviewed · cited 9 times · Not summarized yet

   SCOPE compresses prompts through chunk-level rewriting and summarization instead of token removal.

   Abstract

   A big issue in modern LLM applications is they tend to feed long context to LLM, which results in high inference cost and latency, and may exceed the context limit. Prompt compression addresses this issue by reducing the length of input context with minimum loss of generation quality, i.e, the goal of prompt compression is to shorten the LLM input while maintaining a high generation quality. To overcome these limitations, we propose SCOPE, a training-free generative prompt compression framework based on chunk-level rewriting. Unlike the existing token removal methods, our method centers at a chunking-and-summarization mechanism. Specifically, SCOPE splits a prompt into semantically coherent …

5. Very close Same question Not relevant

   FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attribution

   Syed Rifat Raiyan, M. F. Ishmam, Abdullah Al-Imran et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   FrugalPrompt removes low-utility prompt tokens to reduce inference overhead while preserving the prompt's semantic content.

   Abstract

   Human communication heavily relies on laconism and inferential pragmatics, allowing listeners to successfully reconstruct rich meaning from sparse, telegraphic speech. In contrast, large language models (LLMs) owe much of their stellar performance to expansive input contexts, yet such verbosity inflates monetary costs, carbon footprint, and inference-time latency. This overhead manifests from the redundant low-utility tokens present in typical prompts, as only a fraction of tokens typically carries the majority of the semantic weight. Inspired by the aforementioned cognitive psycholinguistic processes, we address this inefficiency by introducing FrugalPrompt, a novel prompt compression frame…