DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Cité par cet article Pas pertinent

   Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference

   Jovan Stojkovic, Esha Choukse, Chaojie Zhang et al. · 2024 · Artificial Intelligence · preprint, non relu par les pairs · cité 110 fois · Résumé IA disponible

   Résumé scientifique

   With the ubiquitous use of modern large language models (LLMs) across industries, the inference serving for these models is ever expanding. Given the high compute and memory requirements of modern LLMs, more and more top-of-the-line GPUs are being deployed to serve these models. Energy availability has come to the forefront as the biggest challenge for data center expansion to serve these models. In this paper, we present the trade-offs brought up by making energy efficiency the primary goal of LLM serving under performance SLOs. We show that depending on the inputs, the model, and the service-level agreements, there are several knobs available to the LLM inference provider to use for being …

2. Très proche Cite cet article Utilise cet article en profondeur Pas pertinent

   GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving

   Qun-You Liu, Da-Rong Huang, M. Zapater et al. · 2025 · arXiv.org · cité 22 fois · Pas encore résumé

   Résumé scientifique

   Today's Large Language Models (LLMs) inference costs are dominated by GPU energy consumption. LLM inference consists of two stages: prefill and decode, each exhibiting different characteristics. However, current GPU power governors disregard these differences and are not aware of runtime performance, leading to suboptimal DVFS configurations, head-of-line blocking, and excessive energy consumption. We introduce GreenLLM, a service-level objectives (SLO)-aware serving framework that minimizes GPU energy by explicitly separating prefill and decode control with a runtime optimization algorithm for end-to-end energy reduction without sacrificing performance. For prefill, GreenLLM solves a queuei…

3. Très proche Pas pertinent

   Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads

   Grant Wilkins, Srinivasan Keshav, Richard Mortier · 2024 · Energy-Efficient Computing and Networking · cité 43 fois · Pas encore résumé

   Résumé scientifique

   Both the training and use of Large Language Models (LLMs) require large amounts of energy. Their increasing popularity, therefore, raises critical concerns regarding the energy efficiency and sustainability of data centers that host them. This paper addresses the challenge of reducing energy consumption in data centers running LLMs. We propose a hybrid data center model that uses a cost-based scheduling framework to dynamically allocate LLM tasks across hardware accelerators that differ in their energy efficiencies and computational capabilities. Specifically, our workload-aware strategy determines whether tasks are processed on energy-efficient processors or high-performance GPUs based on t…

4. Très proche Pas pertinent

   LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services

   Małgorzata Łazuka, Andreea Anghel, Thomas Parnell · 2024 · Distributed, Parallel, and Cluster Computing · cité 42 fois · Pas encore résumé

   Résumé scientifique

   As Large Language Models (LLMs) are rapidly growing in popularity, LLM inference services must be able to serve requests from thousands of users while satisfying performance requirements. The performance of an LLM inference service is largely determined by the hardware onto which it is deployed, but understanding of which hardware will deliver on performance requirements remains challenging. In this work we present LLM-Pilot - a first-of-its-kind system for characterizing and predicting performance of LLM inference services. LLM-Pilot performs benchmarking of LLM inference services, under a realistic workload, across a variety of GPUs, and optimizes the service configuration for each conside…

5. Très proche Pas pertinent

   UELLM: A Unified and Efficient Approach for LLM Inference Serving

   Yi-Yuan He, Min-Xian Xu, Jing-Feng Wu et al. · 2024 · arXiv.org · preprint, non relu par les pairs · cité 4 fois · Pas encore résumé

   Résumé scientifique

   In the context of Machine Learning as a Service (MLaaS) clouds, the extensive use of Large Language Models (LLMs) often requires efficient management of significant query loads. When providing real-time inference services, several challenges arise. Firstly, increasing the number of GPUs may lead to a decrease in inference speed due to heightened communication overhead, while an inadequate number of GPUs can lead to out-of-memory errors. Secondly, different deployment strategies need to be evaluated to guarantee optimal utilization and minimal inference latency. Lastly, inefficient orchestration of inference queries can easily lead to significant Service Level Objective (SLO) violations. Last…

6. Très proche Cite cet article Pas pertinent

   Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference

   Jia-Ming Cheng, Duong Tung Nguyen · 2025 · cité 3 fois · Pas encore résumé

   Résumé scientifique

   This paper investigates the optimal allocation of large language model (LLM) inference workloads across heterogeneous edge data centers over time. Each data center features on-site renewable generation and faces dynamic electricity prices and spatiotemporal variability in renewable availability. We propose Green-LLM, a lexicographic multi-objective optimization framework that addresses this challenge without requiring manual weight tuning. The proposed model incorporates real-world constraints, including token-dependent processing delay and energy consumption, heterogeneous hardware capabilities, dynamic renewable generation, and spatiotemporal variations in electricity prices and carbon int…

7. Très proche Cité par cet article Utilisé en profondeur par cet article Pas pertinent

   Characterizing Power Management Opportunities for LLMs in the Cloud

   P. Patel, Esha Choukse, Chaojie Zhang et al. · 2024 · International Conference on Architectural Support for Programming Languages and Operating Systems · cité 207 fois · Pas encore résumé

   Résumé scientifique

   Recent innovation in large language models (LLMs), and their myriad use cases have rapidly driven up the compute demand for datacenter GPUs. Several cloud providers and other enterprises plan to substantially grow their datacenter capacity to support these new workloads. A key bottleneck resource in datacenters is power, which LLMs are quickly saturating due to their rapidly increasing model sizes. We extensively characterize the power consumption patterns of a variety of LLMs and their configurations. We identify the differences between the training and inference power consumption patterns. Based on our analysis, we claim that the average and peak power utilization in LLM inference clusters…

8. Très proche Pas pertinent

   Camel: Energy-Aware LLM Inference on Resource-Constrained Devices

   Hao Xu, Long Peng, She-Zheng Song et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 20 fois · Pas encore résumé

   Résumé scientifique

   Most Large Language Models (LLMs) are currently deployed in the cloud, with users relying on internet connectivity for access. However, this paradigm faces challenges such as network latency, privacy concerns, and bandwidth limits. Thus, deploying LLMs on edge devices has become an important research focus. In edge inference, request latency is critical as high latency can impair real-time tasks. At the same time, edge devices usually have limited battery capacity, making energy consumption another major concern. Balancing energy consumption and inference latency is essential. To address this, we propose an LLM inference energy management framework that optimizes GPU frequency and batch size…

Les travaux antérieurs sur lesquels il s’appuie

1. Très proche Pas pertinent

   SLO-Aware GPU DVFS for Energy-Efficient LLM Inference Serving

   A. Kakolyris, D. Masouros, Sotirios Xydis et al. · 2024 · IEEE computer architecture letters · cité 40 fois · Pas encore résumé

   Résumé scientifique

   The increasing popularity of LLM-based chatbots combined with their reliance on power-hungry GPU infrastructure forms a critical challenge for providers: minimizing energy consumption under Service-Level Objectives (SLOs) that ensure optimal user experience. Traditional energy optimization methods fall short for LLM inference due to their autoregressive architecture, which renders them incapable of meeting a predefined SLO without energy overprovisioning. This autoregressive nature however, allows for iteration-level adjustments, enabling continuous fine-tuning of the system throughout the inference process. In this letter, we propose a solution based on iteration-level GPU Dynamic Voltage F…

2. Proche Utilisé en profondeur par cet article Pas pertinent

   Splitwise: Efficient Generative LLM Inference Using Phase Splitting

   P. Patel, Esha Choukse, Chaojie Zhang et al. · 2023 · International Symposium on Computer Architecture · cité 1 078 fois · Pas encore résumé

   Résumé scientifique

   Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwis…

3. Proche Utilisé en profondeur par cet article Pas pertinent

   Efficient Memory Management for Large Language Model Serving with PagedAttention

   Woosuk Kwon, Zhuohan Li, Siyuan Zhuang et al. · 2023 · Machine Learning · cité 9 683 fois · Résumé IA disponible

   Résumé scientifique

   High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within an…

4. Proche Pas pertinent

   Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline

   Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue et al. · 2023 · Neural Information Processing Systems · cité 166 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) have revolutionized the field of AI, demonstrating unprecedented capacity across various tasks. However, the inference process for LLMs comes with significant computational costs. In this paper, we propose an efficient LLM inference pipeline that harnesses the power of LLMs. Our approach begins by tapping into the potential of LLMs to accurately perceive and predict the response length with minimal overhead. By leveraging this information, we introduce an efficient sequence scheduling technique that groups queries with similar response lengths into micro-batches. We evaluate our approach on real-world instruction datasets using the LLaMA-based model, and our resu…

5. Proche Pas pertinent

   Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

   Dujian Ding, Ankur Mallick, Chi Wang et al. · 2024 · International Conference on Learning Representations · cité 514 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our appro…

Les articles qui le citent

1. Très proche Utilise cet article en profondeur Pas pertinent

   Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments

   Nikoleta Iliakopoulou, Jovan Stojkovic, Chloe Alverti et al. · 2024 · Micro · cité 21 fois · Pas encore résumé

   Résumé scientifique

   The effectiveness of LLMs has triggered an exponential rise in their deployment, imposing substantial demands on inference clusters. Such clusters often handle numerous concurrent queries for different LLM downstream tasks. To handle multi-task settings with vast LLM parameter counts, Low-Rank Adaptation (LoRA) enables task-specific fine-tuning while sharing most of the base LLM model across tasks. Hence, it supports concurrent task serving with reduced memory requirements. However, existing designs face inefficiencies: they overlook workload heterogeneity, impose high CPU-GPU link bandwidth from frequent adapter loading, and suffer from head-of-line blocking in their schedulers. To address …

2. Très proche Utilise cet article en profondeur Pas pertinent

   Engine-Agnostic Model Hot-Swapping for Cost-Effective LLM Inference

   Radostin Stoyanov, Viktória Spišaková, Adrian Reber et al. · 2025 · SC25-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis · cité 5 fois · Pas encore résumé

   Résumé scientifique

   The widespread adoption of Large Language Models (LLMs) has led to an increased demand for large-scale inference services, presenting a unique set of challenges for the HPC community. These services are characterized by moderate-scale models that require dedicating expensive GPUs to handle bursty inference requests, leading to high costs and resource underutilization. In this paper, we propose SwapServeLLM — a novel engine-agnostic hot-swapping method for cost-effective inference. This model hot-swapping approach is enabled by recent driver capabilities for transparent GPU checkpointing. SwapServeLLM optimizes resource utilization by dynamically allocating GPU resources with two key mechanis…

3. Très proche Pas pertinent

   Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs

   You-He Jiang, Fang-Cheng Fu, Xiaozhe Yao et al. · 2025 · International Conference on Machine Learning · cité 50 fois · Pas encore résumé

   Résumé scientifique

   Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However, this in turn degrades the cost-efficiency of LLM serving as common practices primarily rely on homogeneous GPU resources. In response to this problem, this work conducts a thorough study about serving LLMs over heterogeneous GPU resources on cloud platforms. The rationale is that different GPU types exhibit distinct compute and memory characteristics, aligning well with the divergent resource demands of diverse requests. Particularly, through comprehensive benchmarking, we discover that the cost-efficiency of LLM…

4. Très proche Utilise cet article en profondeur Pas pertinent

   Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane

   Twinkll Sisodia · 2026 · Pas encore résumé

   Résumé scientifique

   Large language model (LLM) sustainability is increasingly a serving-systems problem, not only a training problem. In production, energy and carbon impact depend on more than model size: workload shape, batching, key-value (KV) cache reuse, prefill/decode placement, model and accelerator choice, power state, geographic carbon intensity, and service-level objectives (SLOs) all matter. Recent systems papers study many of these factors separately. This paper connects those results and asks a practical engineering question: what do they imply when the decision point is a distributed inference control plane such as llm-d? The contribution here is synthesis, not a new set of benchmark results. Repo…

5. Proche Utilise cet article en profondeur Pas pertinent

   FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs

   Xuan He, Ze-Quan Fang, Jinzhao Lian et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 6 fois · Pas encore résumé

   Résumé scientifique

   The ever-increasing computation and energy demand for LLM and AI agents call for holistic and efficient optimization of LLM serving systems. In practice, heterogeneous GPU clusters can be deployed in a geographically distributed manner, while LLM load also observes diversity in terms of both query traffic and serving patterns. LLM queries running on advanced GPUs during a high-emission hour at one location can lead to significantly higher carbon footprints versus same queries running on mid-level GPUs at a low-emission time and location. By observing LLM serving requirements and leveraging spatiotemporal computation flexibility, we consider the joint routing and scheduling problem, and propo…