# Pathways: Asynchronous Distributed Dataflow for ML

**Authors:** Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Dan Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, Brennan Saeta, Parker Schuh, Ryan Sepassi, Laurent El Shafey, Chandramohan A. Thekkath, Yonghui Wu  
**arXiv:** [2203.12533](https://arxiv.org/abs/2203.12533)  
**Date:** 2022-03-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.12533v1/

## Key points

- Pathways is a large-scale orchestration layer designed for accelerators to facilitate exploration of novel systems and machine learning research ideas while maintaining top-notch performance for current models.
- The system utilizes a sharded dataflow graph with asynchronous operators that efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators and coordinates data transfers over dedicated interconnects.
- Pathways features an innovative asynchronous distributed dataflow design that allows the control plane to execute in parallel despite dependencies in the data plane, simplifying the expression of complex new parallelism patterns.
- Demonstrated performance parity with state-of-the-art systems by achieving close to 100% accelerator utilization when running Single Program Multiple Data (SPMD) computations over 2048 Tensor Processing Units (TPUs).
- Pathways delivers throughput comparable to SPMD case for Transformer models pipelined across 16 stages or sharded across two islands of accelerators connected via a data center network.
- The system effectively handles diverse ML workloads, maximizes accelerator utilization, and overall system performance, showcasing potential for driving advancements in machine learning research and cutting-edge model development.

## Summary

In their paper "Pathways: Asynchronous Distributed Dataflow for ML," Barham et al. present the design of a new large-scale orchestration layer for accelerators. The system, Pathways, is specifically crafted to facilitate the exploration of novel systems and machine learning (ML) research ideas while maintaining top-notch performance for current models. <br><br>

Pathways utilizes a sharded dataflow graph comprising asynchronous operators that consume and produce futures. It efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects. A key feature of Pathways is its innovative asynchronous distributed dataflow design, which allows the control plane to execute in parallel despite dependencies in the data plane. This unique design, coupled with meticulous engineering, enables Pathways to adopt a single-controller model that simplifies the expression of complex new parallelism patterns.<br><br>

The authors demonstrate that Pathways can achieve performance parity with state-of-the-art systems by achieving close to 100% accelerator utilization when running Single Program Multiple Data (SPMD) computations over 2048 Tensor Processing Units (TPUs). Furthermore, Pathways delivers throughput comparable to the SPMD case for Transformer models that are either pipelined across 16 stages or sharded across two islands of accelerators connected via a data center network.<br><br>

By showcasing these results, Barham et al. highlight how Pathways can handle diverse ML workloads effectively while maximizing accelerator utilization and overall system performance. In conclusion, "Pathways: Asynchronous Distributed Dataflow for ML" introduces an advanced orchestration layer that not only meets the demands of current ML models but also paves the way for future innovations in parallel computing and system architecture. The system's ability to efficiently manage asynchronous operations and coordinate parallel computations on a large scale demonstrates its potential to drive advancements in machine learning research and accelerate the development of cutting-edge models.