# FastMoE: A Fast Mixture-of-Expert Training System

**Authors:** Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, Jie Tang  
**arXiv:** [2103.13262](https://arxiv.org/abs/2103.13262)  
**Date:** 2021-03-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.13262v1/

## Key points

- Authors discuss the potential of Mixture-of-Expert (MoE) models in scaling up language models to trillions of parameters
- Existing platforms for meeting these requirements rely heavily on Google's hardware and software stack, which are not accessible to the public
- FastMoE is introduced as a distributed MoE training system based on PyTorch with support for common accelerators
- FastMoE offers a hierarchical interface for flexible model design and easy adaptation to various applications
- Training speed is optimized through high-performance acceleration techniques by placing experts on multiple GPUs across multiple nodes
- FastMoE's capabilities and source code are provided in the paper and available under an Apache-2 license
- FastMoE presents an accessible alternative to Google's proprietary platform for training trillion-scale MoE models
- Compatible with PyTorch and common accelerators, making it valuable for researchers and practitioners in GPU and PyTorch communities.

## Summary

In their paper titled "FastMoE: A Fast Mixture-of-Expert Training System," authors Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang discuss the potential of Mixture-of-Expert (MoE) models in scaling up language models to trillions of parameters. To address the limitation that existing platforms capable of meeting these requirements rely heavily on Google's hardware (TPU) and software (Mesh Tensorflow) stack which are not open or accessible to the public, particularly for GPU and PyTorch communities, they introduce FastMoE—a distributed MoE training system based on PyTorch with support for common accelerators. The system offers a hierarchical interface that allows for flexible model design and easy adaptation to various applications like Transformer-XL and Megatron-LM. It optimizes training speed through sophisticated high-performance acceleration techniques by enabling the placement of different experts on multiple GPUs across multiple nodes. The authors provide further details about FastMoE's capabilities in their paper and make its source code available at https://github.com/laekov/fastmoe under an Apache-2 license. Overall, FastMoE presents a promising solution for training trillion-scale MoE models by offering an accessible alternative to Google's proprietary platform. Its compatibility with PyTorch and common accelerators makes it a valuable resource for researchers and practitioners in the GPU and PyTorch communities seeking to leverage MoE models for large-scale language processing tasks.