# Efficiently Programming Large Language Models using SGLang

**Authors:** Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng  
**arXiv:** [2312.07104](https://arxiv.org/abs/2312.07104)  
**Date:** 2023-12-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.07104v1/

## Key points

- SGLang: A domain-specific language embedded in Python for LLM programming
- RadixAttention: A novel technique for automatic KV cache reuse in LLM applications
- Optimization Techniques: Parallelism, batching, caching, sharing, and other compilation techniques in SGLang
- Simplification: Reduced code complexity and enhanced control in LLM programming with SGLang
- Efficient Programming and Execution: Contribution towards efficient programming and execution of Large Language Models using SGLang

## Summary

, , , , 

The paper introduces SGLang, a Structured Generation Language for Large Language Models (LLMs) that aims to improve the efficiency of programming and execution of LLM applications. LLMs are increasingly used for complex tasks involving multiple generation calls, advanced prompting techniques, control flow, and interaction with external environments. However, existing systems lack efficiency in these areas. <kw>SGLang:</kw> A domain-specific language embedded in Python that provides primitives for common LLM programming patterns. <kw>RadixAttention:</kw> A novel technique proposed by the paper that maintains a Least Recently Used (LRU) cache of Key-Value (KV) pairs in a radix tree, enabling automatic KV cache reuse across multiple generation calls at runtime. <kw>Optimization Techniques:</kw> SGLang incorporates parallelism, batching, caching, sharing, and other compilation techniques to improve execution efficiency. <kw>Simplification:</kw> SGLang simplifies the writing of LLM programs and reduces code complexity while enhancing control. <kw>Efficient Programming and Execution:</kw> This research presents an important contribution towards efficient programming and execution of Large Language Models using SGLang.