# Automatic Gradient Descent: Deep Learning without Hyperparameters

**Authors:** Jeremy Bernstein, Chris Mingard, Kevin Huang, Navid Azizan, Yisong Yue  
**arXiv:** [2304.05187](https://arxiv.org/abs/2304.05187)  
**Date:** 2023-04-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.05187v1/

## Key points

- Neural network architecture is crucial for its performance in deep learning
- Existing optimization frameworks often overlook this information
- This paper introduces a new framework that explicitly leverages neural architecture for optimization algorithms
- The theory extends mirror descent to non-convex composite objective functions by transforming a Bregman divergence to account for the non-linear structure of neural architecture
- Automatic gradient descent is obtained - a first-order optimizer without any hyperparameters that can train both fully-connected and convolutional networks out-of-the box and at ImageNet scale
- A PyTorch implementation is available on GitHub and in Appendix B
- This paper provides a rigorous theoretical foundation for next generation architecture-dependent optimizers that work automatically and without hyperparameters, paving the way for more efficient and effective deep learning models.

## Summary

In the field of deep learning, the architecture of a neural network is essential for its performance. Existing optimization frameworks often overlook this information in favor of implicit architectural information or architecture-agnostic distance functions. This paper introduces a new framework to derive optimization algorithms that explicitly leverage neural architecture. The theory extends mirror descent to non-convex composite objective functions by transforming a Bregman divergence to account for the non-linear structure of neural architecture. As a result, automatic gradient descent is obtained - a first-order optimizer without any hyperparameters that can train both fully-connected and convolutional networks out-of-the box and at ImageNet scale. A PyTorch implementation is available on GitHub and in Appendix B. This paper provides a rigorous theoretical foundation for next generation architecture-dependent optimizers that work automatically and without hyperparameters, paving the way for more efficient and effective deep learning models.