# Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

**Authors:** Yunze Wei, Tianshuo Hu, Cong Liang, Yong Cui  
**arXiv:** [2403.07585](https://arxiv.org/abs/2403.07585)  
**Date:** 2024-03-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.07585v1/

## Key points

- Challenges and opportunities surrounding communication optimization in distributed deep neural network training
- Crucial role of communication optimization in overall training time due to increased demand for large-scale models
- Three-layer paradigm: Parallelization Strategy, Collective Communication Library, Network
- Review of current research advances and potential for cross-layer collaborative optimization
- Introduction of a five-layer paradigm emphasizing collaboration designs across layers to improve communication efficiency

## Summary

The article "Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities" by Yunze Wei, Tianshuo Hu, Cong Liang, and Yong Cui delves into the challenges and opportunities surrounding communication optimization in distributed deep neural network training. As the demand for large-scale deep neural network models with increasing parameter numbers continues to grow, so does the need for distributed training due to its requirement of substantial memory and computing resources beyond that of a single GPU. With the advancement of GPU performance leading to decreased computation time, the authors highlight the crucial role of communication optimization in overall training time. They introduce a three-layer paradigm consisting of Parallelization Strategy, Collective Communication Library, and Network to analyze relationships and optimize communication in distributed training. The article also reviews current research advances within this paradigm and identifies potential for cross-layer collaborative optimization. In addition, the authors propose a five-layer paradigm that emphasizes collaboration designs across layers in distributed training scenarios to improve communication efficiency. These include "Vertical", "Horizontal", "Intra-Inter", and "Host-Net" collaboration designs. By shedding light on future research directions in communication optimization for distributed training, this article provides valuable insights into enhancing scalability and efficiency of deep neural network models through optimized communication strategies.