# The Non-IID Data Quagmire of Decentralized Machine Learning

**Authors:** Kevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip B. Gibbons  
**arXiv:** [1910.00189](https://arxiv.org/abs/1910.00189)  
**Date:** 2019-10-01  
**Summary page:** https://www.summarizepaper.com/en/paper/1910.00189v2/

## Key points

- The paper explores challenges posed by datasets generated at different devices and locations in large-scale machine learning applications that require decentralized learning.
- Skewed data labels are a fundamental and pervasive problem for decentralized learning, causing significant accuracy loss across many ML applications, DNN models, training datasets, and decentralized learning algorithms.
- The problem is particularly challenging for DNN models with batch normalization.
- SkewScout is proposed as a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions.
- Group normalization can recover much of the accuracy loss caused by batch normalization.

## Summary

The Non-IID Data Quagmire of Decentralized Machine Learning is a paper authored by Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons that explores the challenges posed by datasets generated at different devices and locations in large-scale machine learning (ML) applications that require decentralized learning. The authors present a detailed experimental study of decentralized deep neural network (DNN) training on a common type of data skew: skewed distribution of data labels across devices/locations. Their study reveals that skewed data labels are a fundamental and pervasive problem for decentralized learning, causing significant accuracy loss across many ML applications, DNN models, training datasets, and decentralized learning algorithms. The problem is particularly challenging for DNN models with batch normalization. Additionally, the degree of data skew is a key determinant of the difficulty of the problem. Based on their findings, the authors propose SkewScout - a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions. They also demonstrate that group normalization can recover much of the accuracy loss caused by batch normalization. The paper highlights how datasets generated from different contexts result in significant data distribution skew across devices/locations and pose a significant challenge to decentralized learning. It provides insights into how this challenge affects various aspects of ML applications and presents solutions to mitigate its impact on accuracy loss during DNN training.