# Benchmarking and Dissecting the Nvidia Hopper GPU Architecture

**Authors:** Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Qiang Wang, Xiaowen Chu  
**arXiv:** [2402.13499](https://arxiv.org/abs/2402.13499)  
**Date:** 2024-02-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.13499v1/

## Key points

- Demand for computational power in GPUs driven by AI and deep learning workloads
- Introduction of Hopper GPUs with unique features like tensor cores supporting FP8, DPX, and distributed shared memory
- Proposal for a benchmarking study focusing on Nvidia Hopper GPU architecture to unravel microarchitectural complexities
- Two key aspects of the study: traditional latency and throughput comparison benchmarks across recent GPU architectures, in-depth exploration of novel features introduced by Hopper GPUs
- Focus on Hopper DPX dynamic programming instruction set, distributed shared memory capabilities, and integration of FP8 tensor cores
- Aim to enhance software optimization strategies and modeling efforts tailored for GPU architectures through deeper understanding of AI function units and programming features in Hopper architecture

## Summary

, , , , 

In the rapidly evolving landscape of graphics processing units (GPUs), the demand for computational power to support modern general-purpose workloads, especially those rooted in artificial intelligence (AI) and deep learning techniques, continues to drive innovation. A significant body of research has been dedicated to analyzing the microarchitectural characteristics of various GPU generations, providing valuable insights for optimizing GPU programs. However, with the introduction of the latest Hopper GPUs, a new frontier emerges with unique features such as tensor cores supporting FP8, DPX, and distributed shared memory. Despite these advancements, there remains a veil of mystery surrounding the performance and operational intricacies of these cutting-edge GPUs. In response to this challenge, a comprehensive benchmarking study focusing on the Nvidia Hopper GPU architecture has been proposed. The primary objective is to unravel the microarchitectural complexities by examining the new instruction-set architecture (ISA) specific to Nvidia GPUs and leveraging new CUDA APIs. This approach encompasses two key aspects: firstly, conducting traditional latency and throughput comparison benchmarks across recent GPU architectures including Hopper, Ada, and Ampere; secondly, delving into an in-depth exploration and benchmarking of the novel features introduced by Hopper GPUs. Specifically, this research delves into the Hopper DPX dynamic programming (DP) instruction set, distributed shared memory capabilities, and integration of FP8 tensor cores. Through meticulous microbenchmarking analysis, a deeper understanding of the unique AI function units and programming features inherent in the Hopper architecture is achieved. This newfound knowledge is poised to significantly enhance software optimization strategies and modeling efforts tailored for GPU architectures. Notably,this study represents a pioneering effort in demystifying performance nuances and programming intricacies associated with tensor cores exclusive to Hopper GPUs. Authored by Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Qiang Wang, and Xiaowen Chu underlines their commitment to advancing our comprehension of cutting-edge GPU technologies for future advancements in AI-driven computing paradigms.