# Prodigy: An Expeditiously Adaptive Parameter-Free Learner

**Authors:** Konstantin Mishchenko, Aaron Defazio  
**arXiv:** [2306.06101](https://arxiv.org/abs/2306.06101)  
**Date:** 2023-06-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.06101v4/

## Key points

- Accurately estimating the learning rate is crucial for optimal performance in adaptive learning methods like AdaGrad and Adam.
- Prodigy algorithm introduced by Konstantin Mishchenko and Aaron Defazio effectively estimates the distance to the solution $D, a key parameter for setting the learning rate optimally.
- Prodigy enhances convergence rate by a factor of $O(\sqrt{\log(D/d_0)})$, where $d_0$ represents the initial estimate of $D, building upon D-Adaptation method for learning-rate-free learning.
- Experiments conducted on various datasets and models show that Prodigy consistently outperforms D-Adaptation and achieves test accuracy values comparable to hand-tuned Adam.
- Prodigy emerges as an expeditiously adaptive parameter-free learner offering significant improvements in estimating the learning rate in adaptive methods, promising enhancements in optimization algorithms for machine learning tasks across domains.

## Summary

, , , , 

In the realm of adaptive learning methods like AdaGrad and Adam, accurately estimating the learning rate is crucial for optimal performance. In this study, authors Konstantin Mishchenko and Aaron Defazio introduce Prodigy, an algorithm designed to effectively estimate the distance to the solution $D$, a key parameter for setting the learning rate optimally. Prodigy builds upon the foundation of the D-Adaptation method for learning-rate-free learning, enhancing its convergence rate by a factor of $O(\sqrt{\log(D/d_0)})$, where $d_0$ represents the initial estimate of $D$. To evaluate its efficacy, experiments were conducted on various datasets and models including 12 common logistic-regression benchmark datasets, VGG11 and ResNet-50 training on CIFAR10, ViT training on Imagenet, LSTM training on IWSLT14, DLRM training on Criteo dataset, VarNet on Knee MRI dataset, as well as RoBERTa and GPT transformer training on BookWiki. The results demonstrate that Prodigy consistently outperforms D-Adaptation and achieves test accuracy values comparable to those achieved by hand-tuned Adam. Overall, Prodigy emerges as an expeditiously adaptive parameter-free learner that offers significant improvements in estimating the learning rate in adaptive methods. This advancement holds promise for enhancing optimization algorithms in machine learning tasks across various domains.