Efficient Memory Management for Large Language Model Serving with PagedAttention

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   Open-AI model Efficient Memory Reduce Management for the Large Language Models (LLMs) Serving with Paged Attention of sharing the KV Cashes

   K. N. Kumar · 2024 · International Journal for Research in Applied Science and Engineering Technology · cited 1 time · Not summarized yet

   Abstract

   Abstract: High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem we proposed a Paged Attention. An alternative algorithm inspired by the classical virtual memory and paging techniques in operating systems. An LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across reque…

2. Very close Not relevant

   vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving

   Jiale Xu, Rui Zhang, Cong Guo et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 25 times · Not summarized yet

   Abstract

   Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. This surge in demand poses significant challenges in optimizing throughput and latency while keeping costs manageable. The Key-Value (KV) cache, a standard method for retaining previous computations, makes LLM inference highly bounded by memory. While batching strategies can enhance performance, they frequently lead to significant memory fragmentation. Even though cutting-edge systems like vLLM mitigate KV cache fragmentation using paged Attention mechanisms, they still suffer from inefficient memory and computational operations due to the tightly coupled page management and computatio…

3. Very close Not relevant

   Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

   Bin Lin, Chen Zhang, Tao Peng et al. · 2024 · Distributed, Parallel, and Cluster Computing · preprint, not peer reviewed · cited 114 times · AI summary available

   Abstract

   Large Language Models (LLMs) demonstrate substantial potential across a diverse array of domains via request serving. However, as trends continue to push for expanding context sizes, the autoregressive nature of LLMs results in highly dynamic behavior of the attention layers, showcasing significant differences in computational characteristics and memory requirements from the non-attention layers. This presents substantial challenges for resource management and performance optimization in service systems. Existing static model parallelism and resource allocation strategies fall short when dealing with this dynamicity. To address the issue, we propose Infinite-LLM, a novel LLM serving system d…

4. Very close Not relevant

   Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

   Bin Gao, Zhuo-Min He, Puru Sharma et al. · 2024 · USENIX Annual Technical Conference · cited 302 times · Not summarized yet

   Abstract

   Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conversations are inefficient due to the need to repeatedly compute the key-value (KV) caches of historical tokens, incurring high serving costs. To address the problem, this paper proposes CachedAttention, a new attention mechanism that enables reuse of KV caches across multi-turn conversations, significantly reducing the repetitive computation overheads. CachedAttention maintains a hierarchical KV caching system that leverages cost-effective memory/storage mediums to save KV caches for all requests. To reduce KV cache …

5. Very close Not relevant

   NanoFlow: Towards Optimal Large Language Model Serving Throughput

   Kan Zhu, Yi-Long Zhao, Liangyu Zhao et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 159 times · Not summarized yet

   Abstract

   Large Language Models (LLMs) have resulted in a surging demand for planet-scale serving systems, where tens of thousands of GPUs continuously serve hundreds of millions of users. Consequently, throughput has emerged as a key metric that determines serving systems' performance. Due to large model sizes and memory-intensive self-attention, LLM serving has been commonly assumed to be memory-bound. Through a detailed analysis, we show that despite having memory-intensive components, end-to-end LLM serving is compute bound for most common workloads and LLMs. Alas, most existing serving engines fall short from optimal compute utilization, because the heterogeneous operations that comprise LLM serv…

6. Very close Not relevant

   Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

   Jiantong Jiang, Pei-Yu Yang, Rui Zhang et al. · 2026 · Annual Meeting of the Association for Computational Linguistics · cited 27 times · Not summarized yet

   Abstract

   Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In this survey, we focus on system-aware KV infrastructure for serving LLMs (abbreviated as sKis). We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). Furthermore, we analyze cross-behavior co-design affinity and behavior-objective links, highli…

7. Very close Not relevant

   eLLM: Elastic Memory Management Framework for Efficient LLM Serving

   Jiale Xu, Rui Zhang, Yi Xiong et al. · 2025 · arXiv.org · cited 8 times · Not summarized yet

   Abstract

   GPU memory management is critical for efficient Large Language Model (LLM) serving. LLM memory usage primarily comprises weights, activations, and KV caches. While weights are static, activations and KV caches exhibit dynamic and unpredictable behavior, posing significant memory management challenges. Modern LLM serving systems address this through a dual-level approach: activations inherit static tensor abstractions from deep learning frameworks, while KV caches employ specialized page-table virtualization (i.e., PagedAttention). Although this reduces KV cache fragmentation, the fundamental isolation between activation and KV cache management prevents memory sharing across these spaces, lea…

8. Very close Not relevant

   Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving

   Shihong Gao, Xin Zhang, Yan-Yan Shen et al. · 2025 · Proc. ACM Manag. Data · cited 18 times · Not summarized yet

   Abstract

   Large language model (LLM) inference serving systems are essential to various LLM-based applications. As demand for LLM services continues to grow, scaling these systems to handle high request rates while meeting latency Service-Level Objectives (SLOs), referred to as effective throughput, becomes critical. However, existing systems often struggle to improve effective throughput, primarily due to a significant decline in Time To First Token (TTFT) SLO attainment. We identify two major causes of this bottleneck: (1) memory-intensive KV cache that limits batch size expansion under GPU memory constraints, and (2) rigid batch composition enforced by the default First-Come-First-Serve scheduling …

Prior work it builds on

1. Related Used in depth by this paper A method it uses Not relevant

   Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

   M. Shoeybi, M. Patwary, Raul Puri et al. · 2019 · arXiv.org · preprint, not peer reviewed · cited 3,361 times · Not summarized yet

   Abstract

   Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We…

2. Related Used in depth by this paper A method it uses Not relevant

   LLaMA: Open and Efficient Foundation Language Models

   Hugo Touvron, Thibaut Lavril, Gautier Izacard et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 22,493 times · AI summary available

   Abstract

   We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

3. Close Not relevant

   High-throughput Generative Inference of Large Language Models with a Single GPU

   Ying Sheng, Lian-Min Zheng, Binhang Yuan et al. · 2023 · International Conference on Machine Learning · cited 1,016 times · Not summarized yet

   Abstract

   The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tenso…

4. Related Not relevant

   OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks

   Benoit Steiner, Mostafa Elhoushi, Jacob Kahn et al. · 2022 · cited 11 times · Not summarized yet

   Abstract

   The size of deep neural networks has grown exponentially in recent years. Unfortunately, hardware devices have not kept pace with the rapidly increasing memory requirements. To cope with this, researchers have turned to techniques such as spilling and recomputation, which increase training time, or reduced precision and model pruning, which can affect model accuracy. We present OLLA, an algorithm that optimizes the lifetime and memory location of the tensors used to train neural networks. Our method reduces the memory usage of existing neural networks, without needing any modification to the models or their training procedures. We formulate the problem as a joint integer linear program (ILP)…

5. Related Used in depth by this paper A method it uses Not relevant

   OPT: Open Pre-trained Transformer Language Models

   Susan Zhang, Stephen Roller, Naman Goyal et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 5,147 times · AI summary available

   Abstract

   Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We…

Papers that cite it

1. Close Uses this paper in depth Not relevant

   ALPS: An Application-Aware, Length-Predictive Scheduler for Efficient LLM Inference at Edge

   Chen Chen, Lei Jiao, Richard Mortier · 2026 · IEEE Transactions on Parallel and Distributed Systems · Not summarized yet

   Abstract

   The deployment of Large Language Models (LLMs) on resource-constrained edge server clusters introduces a critical systems challenge: achieving high throughput and low latency under strict GPU memory and bandwidth limits. Current schedulers are ill-suited for this environment, as they treat requests independently, incurring two major inefficiencies. They waste compute and memory on redundant prefill computations for identical, lengthy system prompts across applications, and their phase-agnostic policies create suboptimal batches that poorly overlap compute-intensive prefills with memory-bound decodes, leading to GPU under-utilization. This paper presents ALPS, an application-aware, length-pre…

2. Close Uses this paper in depth Not relevant

   Breaking Memory Wall for Fast Edge LLM Inference Using Contextual Sparsity

   Zhong-Xiang Wei, Yi-Peng Zhou, Jin Zhao et al. · 2026 · IEEE Transactions on Mobile Computing · Not summarized yet

   Abstract

   Deploying Large Language Models (LLMs) on memory-constrained edge servers to serve requests from mobile devices is challenging due to their substantial resource demands. The Key-Value (KV) cache and Feed-Forward Network (FFN) parameters consume the majority of available memory. However, existing methods typically rely on static memory partitioning for these components. This rigidity leads to critical performance bottlenecks: (1) An insufficient region for neurons causes neuron swapping or direct computation on the CPU, introducing high latency. (2) As the KV cache grows until it exhausts the allocated region, the system must resort to high-latency fallbacks (e.g., KV cache offloading, recomp…

3. Close Not relevant

   An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

   Liang Chen, Haozhe Zhao, Tianyu Liu et al. · 2024 · Computer Vision and Pattern Recognition · cited 159 times · AI summary available

   Abstract

   In this study, we identify the inefficient attention phenomena in Large Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5, QwenVL-Chat and Video-LLaVA. We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs, suggesting a need for a sparser approach compared to textual data handling. To this end, we introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency by learning adaptive attention patterns in early layers and pruning visual tokens in subsequent ones. Our evaluations demonstrate FastV's ability to dramatically reduce computational costs (e.g., a 45 redu…

4. Close Uses this paper in depth Not relevant

   UlaCache: Unified Logical Addressing for Sparse LLM Serving

   Isaac Zhu, Alan Cao, Hugh Yin et al. · 2026 · Not summarized yet

   Abstract

   Long contexts increase the memory cost of the key-value (KV) cache in large language model (LLM) inference. Sparse attention reduces attention computation and memory traffic by selecting historical tokens, but the retained KV cache must remain available for later decoding steps. We present UlaCache, a KV cache management system built around a unified logical address space spanning GPU KV pools and optional CPU memory. Stable logical references let the inference engine allocate and select KV independently of physical placement. Common allocation and sparse-access interfaces support data-parallel (DP) and tensor-parallel (TP) attention. For distributed storage, local GPU buffers retain selecte…

5. Close Not relevant

   CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

   Yuhan Liu, Hanchen Li, Yihua Cheng et al. · 2024 · ACM SIGCOMM Conference (SIGCOMM) · cited 119 times · Not summarized yet

   Abstract

   As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays.