# Decentralized Training of Foundation Models in Heterogeneous Environments

**Authors:** Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, Ce Zhang  
**arXiv:** [2206.01288](https://arxiv.org/abs/2206.01288)  
**Date:** 2022-06-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.01288v4/

## Key points

- Require expensive resources like tens of thousands of GPUs
- Typically trained in specialized clusters with fast interconnects and carefully designed software systems
- Difficulties in obtaining and maintaining these clusters
- Proposed solution: Utilizing decentralized compute resources connected by a heterogeneous network
- First study of training large foundation models with model parallelism in a decentralized regime
- Key technical contribution: Scheduling algorithm for allocating computational "tasklets" to decentralized GPU devices connected by a slow heterogeneous network during training
- Proposed formal cost model to optimize allocation strategy using an efficient evolutionary algorithm
- Extensive experiments conducted using real-world network measurements to validate the approach
- Significant improvements in training speed compared to existing systems like Megatron

## Summary

The summary discusses the challenges of training large foundation models and proposes a solution to address them. These models require expensive resources like tens of thousands of GPUs running continuously for months and are typically trained in specialized clusters with fast interconnects and carefully designed software systems. However, obtaining and maintaining these clusters can be costly and difficult. To overcome this issue, the authors suggest utilizing decentralized compute resources connected by a heterogeneous network. This paper presents the first study of training large foundation models with model parallelism in a decentralized regime. The key technical contribution is a scheduling algorithm that allocates computational "tasklets" to decentralized GPU devices connected by a slow heterogeneous network during training. A formal cost model is also proposed to optimize the allocation strategy using an efficient evolutionary algorithm. Extensive experiments were conducted using real-world network measurements to validate their approach, showing significant improvements in training speed compared to existing systems like Megatron. In conclusion, this paper addresses the challenge of training large foundation models by leveraging decentralized compute resources and proposes a scheduling algorithm and cost model for efficient training.