# Optimizing Distributed Training on Frontier for Large Language Models

**Authors:** Sajal Dash, Isaac Lyngaas, Junqi Yin, Xiao Wang, Romain Egele, Guojing Cong, Feiyi Wang, Prasanna Balaprakash  
**arXiv:** [2312.12705](https://arxiv.org/abs/2312.12705)  
**Date:** 2023-12-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.12705v1/

## Key points

- Large language models (LLMs) have shown remarkable success as foundation models
- Fine-tuning LLMs has proven beneficial for various downstream applications
- Training LLMs with billions of parameters requires significant computational resources
- Efficient distributed strategies such as tensor parallelism, pipeline parallelism, and sharded data parallelism are explored to train a trillion-parameter model on the Frontier exascale supercomputer
- The authors analyze these distributed training techniques individually to determine which ones to use and what associated parameters to select
- Hyperparameter tuning is performed to understand the complex interplay between these techniques
- Optimal strategies for training models of different sizes (22B, 175B, and 1T parameters) are identified
- Achieved 100% weak scaling efficiency for the 175B parameter model and 89% and 87% strong scaling efficiency for the 1T model respectively
- Strategies for distributed training of LLMs based on experimental findings and hyperparameter tuning are presented
- Focus on finding the best strategies to train large models on Frontier by combining tensor parallelism, pipeline parallelism, micro-batch size, and gradient accumulation steps
- Empirical analysis of multiple distribution strategies and valuable observations for training a 22B model are provided
- Hyperparameter tuning results reported for a 175B model and a training recipe devised for both the 175B and 1T models
- Enables state-of-the-art distributed training algorithms on AMD hardware using the ROCM software platform
- Serves as a blueprint for efficient training of LLMs on non-NVIDIA and non-CUDA platforms like the AMD-powered Frontier supercomputer
- Presents an optimized distributed training strategy achieved through hyperparameter search effectively managing GPU memory wall and communication latency.

## Summary

Large language models (LLMs) have shown remarkable success as foundation models and fine-tuning them has proven beneficial for various downstream applications. However, training LLMs with billions of parameters requires significant computational resources. To address this challenge, the authors explore efficient distributed strategies such as tensor parallelism, pipeline parallelism and sharded data parallelism to train a trillion-parameter model on the Frontier exascale supercomputer. The authors analyze these distributed training techniques individually to determine which ones to use and what associated parameters to select. They also perform hyperparameter tuning to understand the complex interplay between these techniques. Through their research efforts, they have identified optimal strategies for training models of different sizes: 22B, 175B and 1T parameters. For the 175B parameter model and 1T model, the authors achieved 100% weak scaling efficiency and 89% and 87% strong scaling efficiency respectively. They present a set of strategies for distributed training of LLMs based on experimental findings and hyperparameter tuning. The authors focus on finding the best strategies to train large models on Frontier by combining tensor parallelism, pipeline parallelism, micro-batch size and gradient accumulation steps. They examine each component in detail to optimize them for Frontier's infrastructure. Their objective is not to achieve the highest possible accuracy but rather to enhance the performance characteristics of training processes on HPC systems. The paper outlines various distribution strategies and evaluates their cost for training large LLMs on Frontier. It provides an empirical analysis of multiple distribution strategies and identifies valuable observations for training a 22B model. The authors also report hyperparameter tuning results for a 175B model and devise a training recipe for both the 175B and 1T models. The contributions of this work include enabling state-of-the-art distributed training algorithms on AMD hardware using the ROCM software platform. It serves as a blueprint for efficient training of LLMs on non-NVIDIA and non-CUDA platforms like the AMD-powered Frontier supercomputer. Additionally, the research presents an optimized distributed training strategy achieved through hyperparameter search effectively managing GPU memory wall and communication latency.