# Uniform Learning in a Deep Neural Network via "Oddball" Stochastic Gradient Descent

**Authors:** Andrew J. R. Simpson  
**arXiv:** [1510.02442](https://arxiv.org/abs/1510.02442)  
**Date:** 2015-10-08  
**Summary page:** https://www.summarizepaper.com/en/paper/1510.02442v1/

## Key points

- Andrew J. R. Simpson explores the assumption of uniform difficulty in learning training examples in deep neural networks.
- Simpson introduces a new approach called "Oddball SGD" (novelty-driven stochastic gradient descent).
- Oddball SGD drives training probabilistically based on the error distribution, with frequency proportional to error magnitude.
- The aim is to enforce uniform error across the entire training set.
- Applying Oddball SGD to a video encoding task in a deep neural network demonstrates its effectiveness in achieving uniform error.
- This challenges conventional assumptions and highlights how adjusting training frequency based on error magnitude can improve performance.
- The research sheds light on an important aspect of deep neural network training and provides a novel technique for addressing variations in difficulty among different training examples.
- The findings have implications for improving learning outcomes and optimizing model performance in various applications involving deep neural networks.

## Summary

In this article titled "Uniform Learning in a Deep Neural Network via 'Oddball' Stochastic Gradient Descent," Andrew J. R. Simpson explores the assumption that training examples in deep neural networks are uniformly difficult to learn. To address this issue, Simpson introduces a new approach called "Oddball SGD" (novelty-driven stochastic gradient descent). This method drives training probabilistically based on the error distribution, where the frequency of training is proportional to the magnitude of the training error. By using Oddball SGD, the aim is to enforce uniform error across the entire training set. To demonstrate its effectiveness, Simpson applies it to a deep neural network tasked with encoding a video and shows that this approach can indeed help achieve uniform error across the training set. This finding challenges conventional assumptions and highlights how adjusting training frequency based on error magnitude can improve overall performance. Overall, this research sheds light on an important aspect of deep neural network training and provides a novel technique for addressing variations in difficulty among different training examples. The findings have implications for improving learning outcomes and optimizing model performance in various applications involving deep neural networks.