# Fairness in Serving Large Language Models

**Authors:** Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica  
**arXiv:** [2401.00588](https://arxiv.org/abs/2401.00588)  
**Date:** 2023-12-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.00588v1/

## Key points

- The paper focuses on ensuring fairness in serving high-demand LLM inference services like ChatGPT and BARD.
- Most major LLM inference services implement request rate limits to prevent one client from dominating the queue, but this can lead to under-utilization of resources and subpar client experiences.
- Serving LLMs introduces complexities due to unpredictable request lengths and unique batching characteristics on parallel accelerators.
- The paper introduces a novel concept of LLM serving fairness based on a cost function considering input and output tokens processed.
- The authors propose a scheduling algorithm called Virtual Token Counter (VTC) for efficient and fair client service.
- Research proves a 2x tight upper bound on service difference between backlogged clients while adhering to work-conserving principles.
- Extensive experiments show superior performance of VTC in ensuring fairness across various conditions, enhancing resource utilization, and improving client experiences.

## Summary

The paper "Fairness in Serving Large Language Models" by Ying Sheng et al. focuses on addressing the challenges of ensuring fairness in serving high-demand LLM inference services such as ChatGPT and BARD. These services cater to a diverse range of requests spanning from short chat conversations to lengthy document reading. To prevent any single client from dominating the request queue and to maintain fairness in processing client requests, most major LLM inference services implement request rate limits. However, this simplistic approach often leads to under-utilization of resources and subpar client experiences when spare capacity is available. The authors highlight that while there is an extensive body of literature on fair scheduling techniques, serving LLMs introduces new complexities due to their unpredictable request lengths and unique batching characteristics on parallel accelerators. In response to these challenges, the paper introduces a novel concept of LLM serving fairness based on a cost function that considers the number of input and output tokens processed. To achieve fairness in serving clients efficiently, the authors propose a groundbreaking scheduling algorithm called the Virtual Token Counter (VTC), which operates on a continuous batching mechanism. One key contribution of this research is proving a 2x tight upper bound on the service difference between two backlogged clients while adhering to work-conserving principles. Through extensive experiments and comparisons with baseline methods, the authors demonstrate the superior performance of VTC in ensuring fairness across various conditions. This innovative approach not only enhances resource utilization but also significantly improves client experiences within high-demand LLM inference services.