Matching Workloads to Blackwell: Power-Resolved Benchmarking of NVIDIA B200 and B300 across HPC and AI Workloads

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Broader context Not relevant

   In-Datacenter Performance Analysis of a Tensor Processing Unit

   Norman P. Jouppi, Cliff Young, Nishant Patil et al. · 2017 · Hardware Architecture · cited 5,584 times · AI summary available

   The TPU study measures datacenter neural-network inference performance and energy, while this paper benchmarks Blackwell GPUs across AI and HPC workloads.

   Abstract

   Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU)---deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithrea…

2. Close Broader context Not relevant

   Benchmarking CPU, GPU, and ML Power-Flow Solvers from 14 to 70,000 Buses

   Benjamin Donnot, Sami Tazi, Eva Boguslawski et al. · 2026 · Distributed, Parallel, and Cluster Computing · AI summary available

   The power-flow study compares CPU, GPU, and learned solvers for grid computation, unlike this paper’s Blackwell hardware and workload benchmarking.

   In short

   Optimized parallel CPU and batched GPU physics solvers match or exceed tested GNN power-flow surrogates in throughput while delivering substantially higher accuracy.

3. Close Same question Not relevant

   From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

   Siddharth Samsi, Dan Zhao, Joseph McDonald et al. · 2023 · Computation and Language · cited 376 times · AI summary available

   From Words to Watts measures the energy costs of LLM inference, while this paper evaluates Llama inference throughput and GPU power.

   Abstract

   Large language models (LLMs) have exploded in popularity due to their new generative capabilities that go far beyond prior state-of-the-art. These technologies are increasingly being leveraged in various domains such as law, finance, and medicine. However, these models carry significant computational challenges, especially the compute and energy costs required for inference. Inference energy costs already receive less attention than the energy costs of training LLMs -- despite how often these large models are called on to conduct inference in reality (e.g., ChatGPT). As these state-of-the-art LLMs see increasing usage and deployment in various domains, a better understanding of their resourc…

4. Close Same question Not relevant

   Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systems

   Prasoon Sinha, Akhil Guliani, Rutwik Jain et al. · 2022 · Distributed, Parallel, and Cluster Computing · cited 47 times · Not summarized yet

   Not All GPUs Are Created Equal studies performance variation among accelerator-rich systems, while this paper compares performance and power across B200 and B300 clusters.

   Abstract

   Scientists are increasingly exploring and utilizing the massive parallelism of general-purpose accelerators such as GPUs for scientific breakthroughs. As a result, datacenters, hyperscalers, national computing centers, and supercomputers have procured hardware to support this evolving application paradigm. These systems contain hundreds to tens of thousands of accelerators, enabling peta- and exa-scale levels of compute for scientific workloads. Recent work demonstrated that power management (PM) can impact application performance in CPU-based HPC systems, even when machines have the same architecture and SKU (stock keeping unit). This variation occurs due to manufacturing variability and th…

5. Close Shared data, tool or method Not relevant

   MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs

   Huaizheng Zhang, Yuanming Li, Wencong Xiao et al. · 2023 · Machine Learning · preprint, not peer reviewed · cited 10 times · Not summarized yet

   MIGPerf provides a benchmark tool and experiments for GPU training and inference, whereas this paper benchmarks workloads on B200 and B300.

   Abstract

   New architecture GPUs like A100 are now equipped with multi-instance GPU (MIG) technology, which allows the GPU to be partitioned into multiple small, isolated instances. This technology provides more flexibility for users to support both deep learning training and inference workloads, but efficiently utilizing it can still be challenging. The vision of this paper is to provide a more comprehensive and practical benchmark study for MIG in order to eliminate the need for tedious manual benchmarking and tuning efforts. To achieve this vision, the paper presents MIGPerf, an open-source tool that streamlines the benchmark study for MIG. Using MIGPerf, the authors conduct a series of experiments,…

6. Close Same method, another problem Not relevant

   Benchmarking and Dissecting the Nvidia Hopper GPU Architecture

   Weile Luo, Ruibo Fan, Zeyu Li et al. · 2024 · Hardware Architecture · cited 110 times · AI summary available

   Benchmarking and Dissecting the Nvidia Hopper GPU Architecture characterizes GPU performance and architecture, a hardware-benchmarking approach also used here for Blackwell.

   Abstract

   Graphics processing units (GPUs) are continually evolving to cater to the computational demands of contemporary general-purpose workloads, particularly those driven by artificial intelligence (AI) utilizing deep learning techniques. A substantial body of studies have been dedicated to dissecting the microarchitectural metrics characterizing diverse GPU generations, which helps researchers understand the hardware details and leverage them to optimize the GPU programs. However, the latest Hopper GPUs present a set of novel attributes, including new tensor cores supporting FP8, DPX, and distributed shared memory. Their details still remain mysterious in terms of performance and operational char…

7. Related Broader context Not relevant

   Efficient Large-Scale Language Model Training on GPU Clusters

   Deepak Narayanan, Mohammad Shoeybi, Jared Casper et al. · 2021 · Computation and Language · cited 1,465 times · AI summary available

   Efficient Large-Scale Language Model Training on GPU Clusters addresses distributed LLM training efficiency, while this paper benchmarks Pangu-Weather training and other workloads on GPUs.

   Abstract

   Large language models have led to state-of-the-art accuracies across a range of tasks. However, training these large models efficiently is challenging for two reasons: a) GPU memory capacity is limited, making it impossible to fit large models on a single GPU or even on a multi-GPU server; and b) the number of compute operations required to train these models can result in unrealistically long training times. New methods of model parallelism such as tensor and pipeline parallelism have been proposed to address these challenges; unfortunately, naive usage leads to fundamental scaling issues at thousands of GPUs due to various reasons, e.g., expensive cross-node communication or idle periods w…

8. Related Same question Not relevant

   DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

   Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri et al. · 2024 · Artificial Intelligence · cited 226 times · AI summary available

   DynamoLLM studies performance and energy efficiency in LLM inference clusters, while this paper measures LLM inference throughput and GPU power.

   Abstract

   The rapid evolution and widespread adoption of generative large language models (LLMs) have made them a pivotal workload in various applications. Today, LLM inference clusters receive a large number of queries with strict Service Level Objectives (SLOs). To achieve the desired performance, these models execute on power-hungry GPUs causing the inference clusters to consume large amount of energy and, consequently, result in excessive carbon emissions. Fortunately, we find that there is a great opportunity to exploit the heterogeneity in inference compute properties and fluctuations in inference workloads, to significantly improve energy-efficiency. However, such a diverse and dynamic environm…