# From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

**Authors:** Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, Vijay Gadepally  
**arXiv:** [2310.03003](https://arxiv.org/abs/2310.03003)  
**Date:** 2023-10-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.03003v1/

## Key points

- Large language models (LLMs) are popular for their advanced generative capabilities, surpassing previous state-of-the-art models.
- LLMs are utilized across various domains like law, finance, and medicine.
- Computational challenges associated with LLMs, particularly in terms of compute and energy costs for inference, are significant.
- Focus on inference energy costs is often overshadowed by training costs in large models like ChatGPT.
- Experiments were conducted to analyze computational and energy utilization of inference with LLMs using the open-sourced pre-trained large language model developed by Meta AI called LLaMA.
- Larger variants of LLaMA typically require multiple high-end GPUs for both training and inference.
- Benchmarking and analysis of inference performance and energy costs were done using NVIDIA V100 & A100 GPUs and two datasets (Alpaca and GSM8K).
- Multi-node, multi-GPU inference with model sharding across up to 32 GPUs was included in the experiments for a comprehensive analysis at scale.
- Comparison was made with results from single node instances using smaller variants of the model as a baseline for comparison.
- The study aims to shed light on compute performance and energy utilization characteristics of LLM inference for cost-savings, efficient hardware usage, and optimal deployment strategies.

## Summary

Large language models (LLMs) have gained immense popularity due to their advanced generative capabilities, surpassing previous state-of-the-art models. These technologies are increasingly being utilized across various domains such as law, finance, and medicine. However, the computational challenges associated with LLMs, particularly in terms of compute and energy costs for inference, are significant. Despite the widespread use of large models like ChatGPT for inference tasks, the focus on inference energy costs is often overshadowed by the attention given to training costs. In this paper, we present the results of experiments conducted to analyze the computational and energy utilization of inference with LLMs using , an open-sourced pre-trained large language model developed by Meta AI. The larger variants of LLaMA typically require multiple high-end GPUs for both training and inference. We benchmarked and analyzed the inference performance and energy costs using two generations of popular GPUs (NVIDIA V100 & A100) and two datasets (Alpaca and GSM8K) to provide insights into resource utilization across different tasks. Our experiments included multi-node, multi-GPU inference with model sharding across up to 32 GPUs, offering a comprehensive analysis at scale that is one of the first in this domain. Additionally, we compared results from single node instances using smaller variants of the model as a baseline for comparison. Through our work, we aim to shed light on the compute performance and energy utilization characteristics of LLM inference to facilitate cost-savings, efficient hardware usage, and optimal strategies for deployment. The landscape of large language models has seen rapid growth in recent years with competing advancements in speed and complexity among AI research groups in both private companies and academic institutions. The development paths of LLMs have evolved significantly since 2017 as illustrated in Fig. 1, showcasing a diverse range of models tailored for specific purposes or use-cases within natural language processing tasks. Overall, our study contributes valuable insights into the computational challenges associated with deploying large language models like LLaMA for real-world applications. By providing detailed analysis on compute performance and energy utilization at scale, we hope to encourage further research, benchmarking efforts, and open dissemination of performance characteristics across different hardware configurations and optimization strategies within the field of large language models.