# FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression

**Authors:** Zhenheng Tang, Xueze Kang, Yiming Yin, Xinglin Pan, Yuxin Wang, Xin He, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Amelie Chi Zhou, Bo Li, Bingsheng He, Xiaowen Chu  
**arXiv:** [2410.12707](https://arxiv.org/abs/2410.12707)  
**Date:** 2024-10-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.12707v1/

## Key points

- System created to address hardware scarcity in training large deep neural networks (DNNs), specifically large language models (LLMs)
- Designed for decentralized training using geo-distributed GPUs across computing clusters or individual devices
- Challenges in system design and efficiency include remote automatic differentiation (RAD), flexible model definitions, heterogeneous software support, low resource utilization due to heterogeneous hardware or the straggler problem, and slow network communication
- DNN model represented as a directed acyclic graph of operators (OP-DAG) for customization without low-level operator implementation concerns
- DAG runtime executor enables RAD without consistent low-level ML framework versions
- Workload estimator and OP-Fence scheduler implemented to increase throughput by grouping devices with similar bandwidths and partitioning the DAG
- AdaTopK compressor proposed to adaptively compress intermediate activations and gradients at slower communication links
- Experiments conducted on three real-world testbeds using 48 GPUs connected via 8 Mbps~10 Gbps networks showed FusionLLM achieving speedups ranging from 1.45 to 9.39 times compared to baseline methods while ensuring convergence under varying conditions

## Summary

The  system was created to address the challenges of hardware scarcity in training large deep neural networks (DNNs), specifically large language models (LLMs). It is designed for decentralized training using geo-distributed GPUs across computing clusters or individual devices. This presents challenges in system design and efficiency, such as remote automatic differentiation (RAD), flexible model definitions and heterogeneous software support, low resource utilization due to heterogeneous hardware or the straggler problem, and slow network communication. To overcome these challenges, the system represents the DNN model as a directed acyclic graph of operators (OP-DAG). Each node represents an operator in the DNNs and edges indicate data dependencies between operators. This design allows for customization of any DNN without worrying about low-level operator implementation. It also enables task scheduling with more fine-grained sub-tasks for additional optimization space. Additionally, a DAG runtime executor can implement RAD without requiring consistent low-level ML framework versions. To improve efficiency, a workload estimator has been implemented along with an OP-Fence scheduler that groups devices with similar bandwidths together and partitions the DAG to increase throughput. An AdaTopK compressor has also been proposed to adaptively compress intermediate activations and gradients at slower communication links. Experiments were conducted on three real-world testbeds using 48 GPUs connected via 8 Mbps~10 Gbps networks to evaluate convergence and efficiency of both the system and algorithms used. Results show that FusionLLM can achieve impressive speedups ranging from 1.45 to 9.39 times compared to baseline methods while ensuring convergence under varying conditions.