# Efficient Memory Management for Large Language Model Serving with PagedAttention

**Authors:** Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica  
**arXiv:** [2309.06180](https://arxiv.org/abs/2309.06180)  
**Date:** 2023-09-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.06180v1/

## Key points

- Research paper presented at SOSP 2023 by Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica
- Proposes PagedAttention algorithm for efficient serving of large language models (LLMs)
- Algorithm inspired by virtual memory and paging techniques in operating systems
- Implemented in vLLM system to minimize waste in key-value cache (KV cache) memory
- Enables flexible sharing of KV cache within and across requests to reduce memory usage
- Demonstrates significant improvements in throughput for popular LLMs compared to state-of-the-art systems
- Achieves near-zero waste in KV cache memory with a 2-4x increase in throughput while maintaining latency
- Sets a new standard for efficient large language model serving with effective resource utilization
- Source code for vLLM available on GitHub at https://github.com/vllm-project/vllm

## Summary

Efficient Memory Management for Large Language Model Serving with PagedAttention is a research paper presented at SOSP 2023 by Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. The paper addresses the challenge of high throughput serving of large language models (LLMs) by proposing PagedAttention, an attention algorithm inspired by virtual memory and paging techniques in operating systems. This algorithm is implemented in vLLM, an LLM serving system that minimizes waste in key-value cache (KV cache) memory and enables flexible sharing of KV cache within and across requests to reduce memory usage. for  with  is a research paper presented at SOSP 2023 by Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng,
Lianmin Zheng,Cody Hao Yu,Joseph E.Gonzalez,Hao Zhang,and Ion Stoica.The paper proposes a novel solution to the challenge of high throughput serving of large language models (LLMs). It introduces , an LLM serving system that utilizes an attention algorithm called . This algorithm draws inspiration from virtual memory and paging techniques used in operating systems. Existing systems struggle with inefficient management of KV cache memory due to its dynamic growth and shrinkage leading to fragmentation and redundant duplication. This limitation hinders batching sufficiently many requests at a time. However,< kd>PagedAttention</ kd >and< kd>vLLM</ kd > address this issue by demonstrating significant improvements in throughput for popular LLMs compared to state-of-the-art systems like FasterTransformer and Orca. The enhancements are particularly notable with longer sequences, larger models, and complex decoding algorithms. The paper highlights the near-zero waste achieved in KV cache memory through< kd>vLLM</ kd >'s approach, showcasing a 2-4x increase in throughput while maintaining the same level of latency. By leveraging memory sharing opportunities effectively and optimizing resource utilization within the system architecture,< kd>vLLM</ kd > sets a new standard for efficient large language model serving. The source code for< kd>vLLM</ kd > is publicly available on GitHub at https://github.com/vllm-project/vllm. is maximized with and , resulting in significant improvements in throughput for large language models. This research paper presents a novel solution to the challenge of high throughput serving by introducing an attention algorithm inspired by virtual memory and paging techniques used in operating systems. With its implementation in  this algorithm minimizes waste in key-value cache (KV cache) memory and enables flexible sharing within and across requests, reducing overall memory usage. Through its effective management of resources,  sets a new standard for efficient large language model serving, showcasing impressive results even with longer sequences, larger models, and complex decoding algorithms.