# No More Pesky Learning Rates

**Authors:** Tom Schaul, Sixin Zhang, Yann LeCun  
**arXiv:** [1206.1106](https://arxiv.org/abs/1206.1106)  
**Date:** 2012-06-06  
**Summary page:** https://www.summarizepaper.com/en/paper/1206.1106v1/

## Key points

- Authors Tom Schaul, Sixin Zhang, and Yann LeCun address tuning learning rates in stochastic gradient descent (SGD) algorithms.
- Performance of SGD heavily relies on adjusting and decreasing learning rates over time.
- Proposed novel method automatically adjusts multiple learning rates to minimize expected error by leveraging local gradient variations.
- Experimentation shows algorithm achieves performance comparable to exhaustive systematic search for optimal settings.
- Eliminates need for manual tuning of learning rates, saving time and effort for practitioners.
- Implications for machine learning researchers and practitioners working with SGD include more efficient and automated adjustments.
- Innovative approach opens up new possibilities for improving model training efficiency and effectiveness in various applications.

## Summary

In their paper titled "No More Pesky Learning Rates," authors Tom Schaul, Sixin Zhang, and Yann LeCun address the critical issue of tuning learning rates in stochastic gradient descent (SGD) algorithms. The performance of SGD heavily relies on how learning rates are adjusted and decreased over time. To tackle this challenge, the authors propose a novel method that automatically adjusts multiple learning rates to minimize the expected error at any given time. This  leverages local gradient variations across samples to make informed decisions about updating learning rates. Through experimentation on a variety of convex and non-convex learning tasks, the authors demonstrate that their proposed algorithm achieves performance levels comparable to those obtained through exhaustive systematic search for optimal settings. Importantly, this  effectively eliminates the need for manual tuning of learning rates, streamlining the optimization process and potentially saving significant time and effort for practitioners in the field. The findings presented in this paper have important implications for machine learning researchers and practitioners working with SGD . By offering a more efficient and automated way to adjust , this work contributes to advancing the state-of-the-art in . The authors' innovative approach opens up new possibilities for improving model training efficiency and effectiveness in various applications across different domains.