# In-Datacenter Performance Analysis of a Tensor Processing Unit

**Authors:** Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami  
**arXiv:** [1704.04760](https://arxiv.org/abs/1704.04760)  
**Date:** 2017-04-16  
**Summary page:** https://www.summarizepaper.com/en/paper/1704.04760v1/

## Key points

- Custom ASIC called Tensor Processing Unit (TPU) evaluated
- TPU designed to accelerate neural network inference phase
- TPU features 65,536 8-bit MAC matrix multiply unit with peak throughput of 92 TeraOps/second (TOPS) and on-chip memory of 28 MiB
- Domain-specific hardware like TPU necessary for major improvements in cost-energy-performance
- Deterministic execution model of TPU better suited for meeting 99th-percentile response-time requirement compared to CPUs and GPUs
- Despite numerous MACs and large memory capacity, TPU remains small and low power due to lack of time-varying optimizations
- TPU outperforms contemporary GPU or CPU by approximately 15X–30X in terms of speed on average
- TOPS/Watt measure of energy efficiency is about 30X–80X higher for TPU compared to GPU and CPU
- Incorporating GPU's GDDR5 memory into TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X that of GPU and 200X that of CPU
- Significant performance advantages demonstrated using TPU over traditional CPUs and GPUs for NN inference tasks in datacenter applications

## Summary

This paper evaluates the performance of a custom ASIC called a Tensor Processing Unit (TPU) that has been deployed in datacenters since 2015. The TPU is designed to accelerate the inference phase of neural networks (NN) and features a 65,536 8-bit MAC matrix multiply unit with a peak throughput of 92 TeraOps/second (TOPS) and a large on-chip memory of 28 MiB. The authors argue that domain-specific hardware, like the TPU, is necessary for achieving major improvements in cost-energy-performance. They highlight that the deterministic execution model of the TPU is better suited for meeting the 99th-percentile response-time requirement of NN applications compared to CPUs and GPUs, which rely on time-varying optimizations such as caches, out-of-order execution, multithreading, multiprocessing, and prefetching. These optimizations may improve average throughput but do not guarantee low latency. Despite having numerous MACs and a large memory capacity, the TPU remains relatively small and low power due to its lack of these time-varying optimizations. To assess its performance, the authors compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU—both contemporaries deployed in the same datacenters. The workload used for evaluation consists of production NN applications (MLPs, CNNs, and LSTMs) written in the high-level TensorFlow framework. The results show that despite some applications experiencing low utilization, on average, the TPU outperforms its contemporary GPU or CPU by approximately 15X–30X in terms of speed. Additionally TOPS/Watt—the measure of energy efficiency—is about 30X–80X higher for the TPU compared to both GPU and CPU. The authors also suggest that incorporating the GPU's GDDR5 memory into the TPU would triple achieved TOPS while raising TOPS/Watt to nearly 70X that of GPU and 200X that of CPU. Overall this study demonstrates significant performance advantages of using a TPU over traditional CPUs and GPUs when accelerating NN inference tasks in datacenter applications. The findings support the idea that domain specific hardware can lead to substantial improvements in cost–energy–performance metrics for such tasks.