# FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines

**Authors:** Jiaao He, Jidong Zhai  
**arXiv:** [2403.11421](https://arxiv.org/abs/2403.11421)  
**Date:** 2024-03-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.11421v1/

## Key points

- Authors propose innovative solutions to improve efficiency and throughput of large language models (LLMs) by addressing high cost
- Introduce use of multiple out-of-chassis remote CPUs for KV-cache and related computations, scaling up memory capacity and bandwidth to better utilize GPU resources
- Develop sequence-level load-stabilizing schedule to minimize idling and optimize workload distribution between CPUs and GPUs
- Adopt model-guided approach to orchestrate CPU-GPU interactions, with aggregated memory bandwidth as key metric in selecting CPUs for optimal performance
- Contributions include novel decomposition of auto-regressive transformer models using near-memory processing over KV-cache with out-of-chassis CPUs for increased throughput
- Sequence-level pipeline schedule balances workload variations in token generation using LLMs
- Create performance model that provides optimal hardware configurations based on different model requirements

## Summary

The authors propose innovative solutions to improve the efficiency and throughput of serving large language models (LLMs) by addressing their high cost. They introduce the use of multiple out-of-chassis remote CPUs for KV-cache and related computations, scaling up memory capacity and bandwidth to better utilize GPU resources. Additionally, they develop a sequence-level load-stabilizing schedule to minimize idling and optimize workload distribution between CPUs and GPUs. A model-guided approach is adopted to orchestrate CPU-GPU interactions, with aggregated memory bandwidth identified as a key metric in selecting CPUs for optimal performance. The contributions include a novel decomposition of auto-regressive transformer models using near-memory processing over KV-cache with out-of-chassis CPUs for increased throughput. A sequence-level pipeline schedule balances workload variations in token generation using LLMs. The authors also create a performance model that provides optimal hardware configurations based on different model requirements. The paper covers background information on LLMs and hardware options, the proposed decomposition approach, system design details addressing heterogeneity challenges, implementation specifics, performance comparisons with other systems, experimental results analysis, discussion on related works, and conclusions. Overall, for serving modern LLMs with affordable GPU resources.