# APIServe: Efficient API Support for Large-Language Model Inferencing

**Authors:** Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, Yiying Zhang  
**arXiv:** [2402.01869](https://arxiv.org/abs/2402.01869)  
**Date:** 2024-02-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.01869v1/

## Key points

- The paper introduces APIServe, a novel LLM inference framework designed for API-augmented LLMs
- APIServe minimizes GPU resource wastage caused by API calls and efficiently allocates saved memory
- Dynamic estimation method proposed for calculating API execution time is highlighted, achieving 93% performance compared to using an oracle
- Evaluation results show that APIServe significantly improves serving throughput and completes more requests per second compared to existing frameworks
- APIServe is implemented on top of vLLM utilizing PagedAttention technique and evaluated against various methods on NVIDIA A100 GPUs in AWS instances and Vicuna models on single and distributed GPU environments

## Summary

The paper "APIServe: Efficient API Support for Large-Language Model Inferencing" addresses the integration of large language models (LLMs) with external tools and APIs to enhance their functionality beyond language-centric tasks. The current LLM inference systems are optimized for standalone use but often treat API calls as new requests, resulting in redundant computations and wasted time. To address this issue, the authors introduce APIServe, a novel LLM inference framework designed specifically for API-augmented LLMs. By minimizing GPU resource wastage caused by API calls and efficiently allocating saved memory, APIServe significantly improves overall serving throughput and completes more requests per second compared to existing frameworks. One key aspect highlighted in the paper is the dynamic estimation method proposed for calculating API execution time. This approach is particularly useful for API types with varying execution times or those not profiled offline. By dynamically adjusting the estimated API time based on request pause duration, APIServe achieves 93% performance compared to using an oracle for precise execution time prediction. In terms of evaluation results, APIServe is implemented on top of vLLM utilizing PagedAttention technique and compared against vanilla vLLM, ImprovedDiscard, Preserve, and Swap methods. The evaluation includes augmenting GPT-J models with APIs on NVIDIA A100 GPUs in AWS instances and Vicuna models on both single and distributed GPU environments. Real-world serving scenarios are simulated to assess performance under diverse request types. Overall, the findings demonstrate that APIServe's innovative approach effectively optimizes API support for LLM inferencing. It showcases substantial improvements in efficiency and throughput compared to existing frameworks. This research contributes valuable insights into enhancing the integration of external tools and APIs with large language models for enhanced performance in various applications.