# The AdEMAMix Optimizer: Better, Faster, Older

**Authors:** Matteo Pagliardini, Pierre Ablin, David Grangier  
**arXiv:** [2409.03137](https://arxiv.org/abs/2409.03137)  
**Date:** 2024-09-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.03137v1/

## Key points

- Momentum-based optimizers in machine learning use Exponential Moving Average (EMA) of gradients to adjust contribution of older gradients exponentially
- Conventional single EMA approach for accumulating past gradients is suboptimal due to inability to balance immediate and significantly older gradients effectively
- AdEMAMix optimizer, a modified version of Adam, incorporates mixture of two EMAs for better leveraging past gradient information
- Gradients can remain relevant for tens of thousands of steps, leading to faster convergence and lower minima with AdEMAMix
- AdEMAMix helps in slowing down model forgetting during training and efficiently trains large language models and Vision Transformers (ViTs)
- Proposed optimizer combines slow momentum for gathering information over many timesteps and fast momentum for adapting to rapidly changing loss landscapes

## Summary

, , , , 

In their study titled "The AdEMAMix Optimizer: Better, Faster, Older," researchers Matteo Pagliardini, Pierre Ablin, and David Grangier delve into the realm of momentum-based optimizers in machine learning applications. These optimizers utilize an Exponential Moving Average (EMA) of gradients to adjust the contribution of older gradients exponentially. This adjustment takes into account the fact that gradients serve as local linear approximations that lose relevance as the model iterates through the loss landscape. The research challenges the conventional use of a single EMA to accumulate past gradients and presents empirical evidence showcasing its suboptimal nature. It is argued that a single EMA cannot effectively balance giving high weight to immediate past gradients while also considering significantly older ones. Building on this observation, the researchers introduce AdEMAMix, a modified version of the Adam optimizer that incorporates a mixture of two EMAs to better leverage past gradient information. Through experiments in language modeling and image classification, surprising results emerge indicating that gradients can remain relevant for tens of thousands of steps. The AdEMAMix optimizer not only aids in faster convergence but also often leads to lower minima. For instance, a 1.3 billion parameter AdEMAMix LLM trained on 101 billion tokens performs comparably to an AdamW model trained on 197 billion tokens (a 95% improvement). Additionally, AdEMAMix proves effective in slowing down model forgetting during training. The study concludes by highlighting how leveraging old gradients efficiently trains large language models and Vision Transformers (ViTs). The proposed optimizer combines two momentum terms - a slow momentum for gathering information over many timesteps and a fast momentum for adapting to rapidly changing loss landscapes. Through various experiments in text modeling and image classification, the superiority of AdEMAMix over AdamW is demonstrated, emphasizing its ability to retain training data at a slower pace. This work encourages further exploration into alternative methods beyond EMAs for leveraging past gradient information in optimization algorithms.