DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Same question Cited by this paper Not relevant

   Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference

   Jovan Stojkovic, Esha Choukse, Chaojie Zhang et al. · 2024 · Artificial Intelligence · preprint, not peer reviewed · cited 110 times · AI summary available

   Towards Greener LLMs examines energy-efficiency trade-offs for LLM serving under performance SLOs, as DynamoLLM does.

   Abstract

   With the ubiquitous use of modern large language models (LLMs) across industries, the inference serving for these models is ever expanding. Given the high compute and memory requirements of modern LLMs, more and more top-of-the-line GPUs are being deployed to serve these models. Energy availability has come to the forefront as the biggest challenge for data center expansion to serve these models. In this paper, we present the trade-offs brought up by making energy efficiency the primary goal of LLM serving under performance SLOs. We show that depending on the inputs, the model, and the service-level agreements, there are several knobs available to the LLM inference provider to use for being …

2. Very close Another approach to the problem Cites this paper Uses this paper in depth Not relevant

   GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving

   Qun-You Liu, Da-Rong Huang, M. Zapater et al. · 2025 · arXiv.org · cited 22 times · Not summarized yet

   GreenLLM reduces GPU energy under serving SLOs by separately controlling prefill and decode frequencies, rather than reconfiguring the whole cluster.

   Abstract

   Today's Large Language Models (LLMs) inference costs are dominated by GPU energy consumption. LLM inference consists of two stages: prefill and decode, each exhibiting different characteristics. However, current GPU power governors disregard these differences and are not aware of runtime performance, leading to suboptimal DVFS configurations, head-of-line blocking, and excessive energy consumption. We introduce GreenLLM, a service-level objectives (SLO)-aware serving framework that minimizes GPU energy by explicitly separating prefill and decode control with a runtime optimization algorithm for end-to-end energy reduction without sacrificing performance. For prefill, GreenLLM solves a queuei…

3. Very close Another approach to the problem Not relevant

   Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads

   Grant Wilkins, Srinivasan Keshav, Richard Mortier · 2024 · Energy-Efficient Computing and Networking · cited 43 times · Not summarized yet

   Hybrid Heterogeneous Clusters schedules LLM tasks across energy-efficient processors and high-performance GPUs to reduce data-center energy.

   Abstract

   Both the training and use of Large Language Models (LLMs) require large amounts of energy. Their increasing popularity, therefore, raises critical concerns regarding the energy efficiency and sustainability of data centers that host them. This paper addresses the challenge of reducing energy consumption in data centers running LLMs. We propose a hybrid data center model that uses a cost-based scheduling framework to dynamically allocate LLM tasks across hardware accelerators that differ in their energy efficiencies and computational capabilities. Specifically, our workload-aware strategy determines whether tasks are processed on energy-efficient processors or high-performance GPUs based on t…

4. Very close Broader context Not relevant

   LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services

   Małgorzata Łazuka, Andreea Anghel, Thomas Parnell · 2024 · Distributed, Parallel, and Cluster Computing · cited 42 times · Not summarized yet

   LLM-Pilot benchmarks and predicts inference performance across GPUs to optimize service configurations, rather than targeting energy management.

   Abstract

   As Large Language Models (LLMs) are rapidly growing in popularity, LLM inference services must be able to serve requests from thousands of users while satisfying performance requirements. The performance of an LLM inference service is largely determined by the hardware onto which it is deployed, but understanding of which hardware will deliver on performance requirements remains challenging. In this work we present LLM-Pilot - a first-of-its-kind system for characterizing and predicting performance of LLM inference services. LLM-Pilot performs benchmarking of LLM inference services, under a realistic workload, across a variety of GPUs, and optimizes the service configuration for each conside…

5. Very close Broader context Not relevant

   UELLM: A Unified and Efficient Approach for LLM Inference Serving

   Yi-Yuan He, Min-Xian Xu, Jing-Feng Wu et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 4 times · Not summarized yet

   UELLM addresses GPU deployment, inference latency, and query orchestration for SLO-compliant LLM serving.

   Abstract

   In the context of Machine Learning as a Service (MLaaS) clouds, the extensive use of Large Language Models (LLMs) often requires efficient management of significant query loads. When providing real-time inference services, several challenges arise. Firstly, increasing the number of GPUs may lead to a decrease in inference speed due to heightened communication overhead, while an inadequate number of GPUs can lead to out-of-memory errors. Secondly, different deployment strategies need to be evaluated to guarantee optimal utilization and minimal inference latency. Lastly, inefficient orchestration of inference queries can easily lead to significant Service Level Objective (SLO) violations. Last…

6. Very close Another approach to the problem Cites this paper Not relevant

   Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference

   Jia-Ming Cheng, Duong Tung Nguyen · 2025 · cited 3 times · Not summarized yet

   Green-LLM allocates inference workloads across edge data centers according to renewable availability and electricity prices to address energy and carbon objectives.

   Abstract

   This paper investigates the optimal allocation of large language model (LLM) inference workloads across heterogeneous edge data centers over time. Each data center features on-site renewable generation and faces dynamic electricity prices and spatiotemporal variability in renewable availability. We propose Green-LLM, a lexicographic multi-objective optimization framework that addresses this challenge without requiring manual weight tuning. The proposed model incorporates real-world constraints, including token-dependent processing delay and energy consumption, heterogeneous hardware capabilities, dynamic renewable generation, and spatiotemporal variations in electricity prices and carbon int…

7. Very close Groundwork for this paper Cited by this paper Used in depth by this paper Not relevant

   Characterizing Power Management Opportunities for LLMs in the Cloud

   P. Patel, Esha Choukse, Chaojie Zhang et al. · 2024 · International Conference on Architectural Support for Programming Languages and Operating Systems · cited 207 times · Not summarized yet

   Characterizing Power Management Opportunities measures LLM power patterns and identifies power-management opportunities in inference clusters.

   Abstract

   Recent innovation in large language models (LLMs), and their myriad use cases have rapidly driven up the compute demand for datacenter GPUs. Several cloud providers and other enterprises plan to substantially grow their datacenter capacity to support these new workloads. A key bottleneck resource in datacenters is power, which LLMs are quickly saturating due to their rapidly increasing model sizes. We extensively characterize the power consumption patterns of a variety of LLMs and their configurations. We identify the differences between the training and inference power consumption patterns. Based on our analysis, we claim that the average and peak power utilization in LLM inference clusters…

8. Very close Same method, another problem Not relevant

   Camel: Energy-Aware LLM Inference on Resource-Constrained Devices

   Hao Xu, Long Peng, She-Zheng Song et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 20 times · Not summarized yet

   Camel manages GPU frequency and batch size to balance energy and latency for LLM inference on resource-constrained devices.

   Abstract

   Most Large Language Models (LLMs) are currently deployed in the cloud, with users relying on internet connectivity for access. However, this paradigm faces challenges such as network latency, privacy concerns, and bandwidth limits. Thus, deploying LLMs on edge devices has become an important research focus. In edge inference, request latency is critical as high latency can impair real-time tasks. At the same time, edge devices usually have limited battery capacity, making energy consumption another major concern. Balancing energy consumption and inference latency is essential. To address this, we propose an LLM inference energy management framework that optimizes GPU frequency and batch size…

Prior work it builds on

1. Very close Another approach to the problem Not relevant

   SLO-Aware GPU DVFS for Energy-Efficient LLM Inference Serving

   A. Kakolyris, D. Masouros, Sotirios Xydis et al. · 2024 · IEEE computer architecture letters · cited 40 times · Not summarized yet

   SLO-Aware GPU DVFS adjusts GPU voltage and frequency at each inference iteration to reduce energy while meeting latency SLOs.

   Abstract

   The increasing popularity of LLM-based chatbots combined with their reliance on power-hungry GPU infrastructure forms a critical challenge for providers: minimizing energy consumption under Service-Level Objectives (SLOs) that ensure optimal user experience. Traditional energy optimization methods fall short for LLM inference due to their autoregressive architecture, which renders them incapable of meeting a predefined SLO without energy overprovisioning. This autoregressive nature however, allows for iteration-level adjustments, enabling continuous fine-tuning of the system throughout the inference process. In this letter, we propose a solution based on iteration-level GPU Dynamic Voltage F…

2. Close Groundwork for this paper Used in depth by this paper Not relevant

   Splitwise: Efficient Generative LLM Inference Using Phase Splitting

   P. Patel, Esha Choukse, Chaojie Zhang et al. · 2023 · International Symposium on Computer Architecture · cited 1,078 times · Not summarized yet

   Splitwise uses the differing compute and power characteristics of prompt processing and token generation to split LLM inference across GPUs.

   Abstract

   Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwis…

3. Close Groundwork for this paper Used in depth by this paper Not relevant

   Efficient Memory Management for Large Language Model Serving with PagedAttention

   Woosuk Kwon, Zhuohan Li, Siyuan Zhuang et al. · 2023 · Machine Learning · cited 9,683 times · AI summary available

   PagedAttention and vLLM manage KV-cache memory to enable larger batches and high-throughput LLM serving.

   Abstract

   High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within an…

4. Close Broader context Not relevant

   Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline

   Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue et al. · 2023 · Neural Information Processing Systems · cited 166 times · Not summarized yet

   Response Length Perception predicts output lengths and schedules similar queries together to improve inference efficiency.

   Abstract

   Large language models (LLMs) have revolutionized the field of AI, demonstrating unprecedented capacity across various tasks. However, the inference process for LLMs comes with significant computational costs. In this paper, we propose an efficient LLM inference pipeline that harnesses the power of LLMs. Our approach begins by tapping into the potential of LLMs to accurately perceive and predict the response length with minimal overhead. By leveraging this information, we introduce an efficient sequence scheduling technique that groups queries with similar response lengths into micro-batches. We evaluate our approach on real-world instruction datasets using the LLaMA-based model, and our resu…

5. Close Broader context Not relevant

   Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

   Dujian Ding, Ankur Mallick, Chi Wang et al. · 2024 · International Conference on Learning Representations · cited 514 times · Not summarized yet

   Hybrid LLM routes queries between small and large models to trade response quality against inference cost.

   Abstract

   Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our appro…

Papers that cite it

1. Very close Broader context Uses this paper in depth Not relevant

   Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments

   Nikoleta Iliakopoulou, Jovan Stojkovic, Chloe Alverti et al. · 2024 · Micro · cited 21 times · Not summarized yet

   Chameleon adapts caching and scheduling for concurrent LoRA tasks in inference clusters, addressing workload heterogeneity and blocking.

   Abstract

   The effectiveness of LLMs has triggered an exponential rise in their deployment, imposing substantial demands on inference clusters. Such clusters often handle numerous concurrent queries for different LLM downstream tasks. To handle multi-task settings with vast LLM parameter counts, Low-Rank Adaptation (LoRA) enables task-specific fine-tuning while sharing most of the base LLM model across tasks. Hence, it supports concurrent task serving with reduced memory requirements. However, existing designs face inefficiencies: they overlook workload heterogeneity, impose high CPU-GPU link bandwidth from frequent adapter loading, and suffer from head-of-line blocking in their schedulers. To address …

2. Very close Broader context Uses this paper in depth Not relevant

   Engine-Agnostic Model Hot-Swapping for Cost-Effective LLM Inference

   Radostin Stoyanov, Viktória Spišaková, Adrian Reber et al. · 2025 · SC25-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis · cited 5 times · Not summarized yet

   SwapServeLLM hot-swaps models to allocate GPU resources dynamically and reduce the cost of bursty inference services.

   Abstract

   The widespread adoption of Large Language Models (LLMs) has led to an increased demand for large-scale inference services, presenting a unique set of challenges for the HPC community. These services are characterized by moderate-scale models that require dedicating expensive GPUs to handle bursty inference requests, leading to high costs and resource underutilization. In this paper, we propose SwapServeLLM — a novel engine-agnostic hot-swapping method for cost-effective inference. This model hot-swapping approach is enabled by recent driver capabilities for transparent GPU checkpointing. SwapServeLLM optimizes resource utilization by dynamically allocating GPU resources with two key mechanis…

3. Very close Same question Not relevant

   Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs

   You-He Jiang, Fang-Cheng Fu, Xiaozhe Yao et al. · 2025 · International Conference on Machine Learning · cited 50 times · Not summarized yet

   Demystifying Cost-Efficiency studies how heterogeneous GPUs can match diverse LLM requests to improve serving cost-efficiency.

   Abstract

   Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However, this in turn degrades the cost-efficiency of LLM serving as common practices primarily rely on homogeneous GPU resources. In response to this problem, this work conducts a thorough study about serving LLMs over heterogeneous GPU resources on cloud platforms. The rationale is that different GPU types exhibit distinct compute and memory characteristics, aligning well with the divergent resource demands of diverse requests. Particularly, through comprehensive benchmarking, we discover that the cost-efficiency of LLM…

4. Very close Review Uses this paper in depth Not relevant

   Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane

   Twinkll Sisodia · 2026 · Not summarized yet

   Toward Sustainable Distributed LLM Inference synthesizes work on energy, carbon, caching, placement, and SLO-aware distributed serving.

   Abstract

   Large language model (LLM) sustainability is increasingly a serving-systems problem, not only a training problem. In production, energy and carbon impact depend on more than model size: workload shape, batching, key-value (KV) cache reuse, prefill/decode placement, model and accelerator choice, power state, geographic carbon intensity, and service-level objectives (SLOs) all matter. Recent systems papers study many of these factors separately. This paper connects those results and asks a practical engineering question: what do they imply when the decision point is a distributed inference control plane such as llm-d? The contribution here is synthesis, not a new set of benchmark results. Repo…

5. Close Another approach to the problem Uses this paper in depth Not relevant

   FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs

   Xuan He, Ze-Quan Fang, Jinzhao Lian et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet

   FREESH jointly routes and schedules LLM queries across heterogeneous GPUs and locations to improve fairness, resource efficiency, and energy outcomes.

   Abstract

   The ever-increasing computation and energy demand for LLM and AI agents call for holistic and efficient optimization of LLM serving systems. In practice, heterogeneous GPU clusters can be deployed in a geographically distributed manner, while LLM load also observes diversity in terms of both query traffic and serving patterns. LLM queries running on advanced GPUs during a high-emission hour at one location can lead to significantly higher carbon footprints versus same queries running on mid-level GPUs at a low-emission time and location. By observing LLM serving requirements and leveraging spatiotemporal computation flexibility, we consider the joint routing and scheduling problem, and propo…