# It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization

**Authors:** Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni  
**arXiv:** [2504.13173](https://arxiv.org/abs/2504.13173)  
**Date:** 2025-04-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2504.13173v1/

## Key points

- Authors explore efficient architectural backbones inspired by human cognitive attentional bias
- Introduce alternative attentional bias configurations to stabilize training procedures
- Reinterpret forgetting mechanisms as retention regularization, introducing novel forget gates
- Present Miras framework for designing deep learning architectures based on associative memory architecture, attentional bias objective, retention gate, and memory learning algorithm
- Introduce three novel sequence models—Moneta, Yaad, and Memora—that outperform existing linear RNNs in performance
- Experimental results show varying strengths of models within Miras based on design choices
- Models excel in tasks such as language modeling, commonsense reasoning, and recall-intensive activities
- Evaluation of scaling pattern shows all variants of Miras outperform baselines with increased context length due to expressive memory architecture

## Summary

, , , , 

In their paper "It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization," authors Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni explore efficient architectural backbones to enhance foundation models. They draw inspiration from human cognitive attentional bias to reconceptualize neural architectures like Transformers and Titans as associative memory modules that learn key-value mappings through an internal objective called attentional bias. The authors observe that existing sequence models primarily use dot-product similarity or L2 regression objectives for their attentional bias. To address this limitation, they introduce alternative attentional bias configurations and their approximations to stabilize training procedures. Additionally, they reinterpret forgetting mechanisms in deep learning architectures as retention regularization, introducing novel forget gates for sequence models. Building on these insights, the authors present Miras—a framework for designing deep learning architectures based on choices of associative memory architecture, attentional bias objective, retention gate, and memory learning algorithm. They introduce three novel sequence models—Moneta, Yaad, and Memora—that surpass existing linear RNNs in performance while maintaining fast parallelizable training processes. Experimental results demonstrate that different design choices within Miras yield models with varying strengths. Certain instances of Miras excel in tasks such as language modeling, commonsense reasoning, and recall-intensive activities—outperforming even Transformers and other modern linear recurrent models. Furthermore,<kgd> the authors evaluate the scaling pattern of their models by varying model size and context window length</kgd>. Results show that all variants of Miras outperform state-of-the-art baselines when increasing context length due to its expressive memory architecture. Overall, this paper provides a comprehensive exploration of test-time memorization mechanisms in deep learning architectures and offers a promising framework for designing efficient neural network structures with improved performance across various tasks.