# PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

**Authors:** Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, Shen Li  
**arXiv:** [2304.11277](https://arxiv.org/abs/2304.11277)  
**Date:** 2023-04-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.11277v2/

## Key points

- Large models have the potential to deliver exceptional performance across various domains in machine learning systems research.
- PyTorch Fully Sharded Data Parallel (FSDP) is introduced as an industry-grade solution for large model training, addressing the technical barrier for broader community utilization.
- FSDP is designed in collaboration with key PyTorch core components to ensure seamless user experiences and optimal training efficiency.
- FSDP incorporates techniques and settings that enhance resource utilization across different hardware configurations.
- Experimental validation shows that FSDP can achieve performance levels comparable to Distributed Data Parallel while enabling support for significantly larger models with near-linear scalability in terms of TFLOPS.
- The collaborative efforts of multiple authors have resulted in a groundbreaking advancement that opens up new possibilities for leveraging large models in machine learning applications, promising democratized access to cutting-edge technologies and driving advancements in artificial intelligence.

## Summary

In the realm of machine learning systems research, the potential of large models to deliver exceptional performance across various domains is widely recognized. However, access to these capabilities has been limited to a select group of advanced users and industry leaders, creating a technical barrier for broader community utilization. Addressing this challenge, this paper introduces PyTorch Fully Sharded Data Parallel (FSDP) as an industry-grade solution tailored for large model training. FSDP is meticulously designed in collaboration with key PyTorch core components such as Tensor implementation, dispatcher system, and CUDA memory caching allocator to ensure seamless user experiences and optimal training efficiency. Moreover, FSDP incorporates a diverse array of techniques and settings that enhance resource utilization across different hardware configurations. Through experimental validation, it has been demonstrated that FSDP can achieve performance levels comparable to Distributed Data Parallel while enabling support for significantly larger models with near-linear scalability in terms of TFLOPS. The collaborative efforts of authors Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott,
Sam Shleifer,
Alban Desmaison,
Can Balioglu,
Pritam Damania,
Bernard Nguyen,
Geeta Chauhan,
Yuchen Hao,
Ajit Mathews,
Shen Li have resulted in a groundbreaking advancement that opens up new possibilities for leveraging large models in machine learning applications. This innovative approach holds promise for democratizing access to cutting-edge technologies and driving advancements in the field of artificial intelligence.