# Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

**Authors:** Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica  
**arXiv:** [2201.12023](https://arxiv.org/abs/2201.12023)  
**Date:** 2022-01-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.12023v1/

## Key points

- Alpa is a system that automates the model-parallel training of large deep learning (DL) models
- Alpa generates execution plans that unify data, operator, and pipeline parallelism
- Alpa views parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms
- Alpa constructs a new hierarchical space for massive model-parallel execution plans
- Alpa designs a number of compilation passes to automatically derive the optimal parallel execution plan in each independent parallelism level and implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices
- The evaluation shows that Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for.
- Unlike specialized systems, Alpa generalizes to models with heterogeneous architectures and models without manually designed plans.
- The authors of Alpa are Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo Joseph E. Gonzalez and Ion Stoica.
- Overall, Alpa is a promising system that has the potential to significantly improve the efficiency and scalability of deep learning training on distributed compute devices.

## Summary

Alpa is a system that automates the model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Unlike existing model-parallel training systems which require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations, Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. This approach allows Alpa to construct a new hierarchical space for massive model-parallel execution plans. To achieve this, Alpa designs a number of compilation passes to automatically derive the optimal parallel execution plan in each independent parallelism level and implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. The evaluation shows that Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Moreover, unlike specialized systems, Alpa generalizes to models with heterogeneous architectures and models without manually designed plans. The authors of Alpa are Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo Joseph E. Gonzalez and Ion Stoica. Their paper titled "Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning" presents an innovative solution to the problem of scaling out complex DL models on distributed compute devices by automating inter- and intra-operator parallelism. Overall, Alpa is a promising system that has the potential to significantly improve the efficiency and scalability of deep learning training on distributed compute devices.