# Scaling Exponents Across Parameterizations and Optimizers

**Authors:** Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, Jeffrey Pennington  
**arXiv:** [2407.05872](https://arxiv.org/abs/2407.05872)  
**Date:** 2024-07-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.05872v1/

## Key points

- The paper explores scaling models from small to large width by adjusting algorithmic and architectural details such as parameterization and optimizer choices.
- The authors challenge key assumptions in prior work regarding the alignment between parameters and data, proposing a new perspective on parameterization.
- Through empirical investigations involving tens of thousands of models trained with various optimizers, parameterizations, alignment assumptions, learning rates, and model sizes up to 26.8B parameters, the authors derive new theoretical results under weaker assumptions.
- Findings reveal that the best learning rate scaling prescription may have been overlooked in previous studies due to restrictive assumptions.
- All parameterizations can achieve hyperparameter transfer contrary to existing beliefs that only maximal update parameterization (muP) can achieve this feat.
- Introduction of a novel per-layer learning rate prescription for standard parameterization that outperforms muP is highlighted.
- Importance of correctly scaling the epsilon parameter in Adam optimizer to prevent gradient underflow is emphasized.
- Proposal of Adam-atan2 as a new numerically stable version of Adam that eliminates the epsilon hyperparameter entirely is made.
- The study's comprehensive analysis sheds light on the intricate relationship between parameterization choices and optimizer performance across various model sizes, contributing valuable insights to machine learning scalability.

## Summary

In their paper titled "Scaling Exponents Across Parameterizations and Optimizers," Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, and Jeffrey Pennington explore the robust and effective scaling of models from small to large width by adjusting algorithmic and architectural details such as parameterization and optimizer choices. They propose a new perspective on parameterization by challenging key assumptions in prior work regarding the alignment between parameters and data. Through empirical investigations involving tens of thousands of models trained with various optimizers, parameterizations, alignment assumptions, learning rates, and model sizes up to 26.8B parameters,the authors derive new theoretical results under weaker assumptions. Their findings reveal that the best learning rate scaling prescription may have been overlooked in previous studies due to restrictive assumptions. Contrary to existing beliefs that only maximal update parameterization (muP) can achieve hyperparameter transfer,the authors demonstrate that all parameterizations can achieve this feat. Additionally, they introduce a novel per-layer learning rate prescription for standard parameterization that outperforms muP.Furthermore,the researchers highlight the importance of correctly scaling the epsilon parameter in Adam optimizer to prevent gradient underflow.They propose Adam-atan2 as a new numerically stable version of Adam that eliminates the epsilon hyperparameter entirely. The study's comprehensive analysis sheds light on the intricate relationship between parameterization choices and optimizer performance across various model sizes.By challenging conventional wisdom and introducing innovative approaches to hyperparameter tuningand optimization techniques,this research contributes valuable insights to the field of machine learning scalability.