# Nested Learning: The Illusion of Deep Learning Architectures

**Authors:** Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni  
**arXiv:** [2512.24695](https://arxiv.org/abs/2512.24695)  
**Date:** 2025-12-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2512.24695v1/

## Key points

- Nested Learning (NL) is introduced as a new learning paradigm for machine learning models with nested, multi-level, and/or parallel optimization problems.
- NL aims to address challenges such as continual learning, memorization, self-improvement, and finding effective solutions in deep learning models.
- Gradient-based optimizers like Adam and SGD with Momentum function as associative memory modules that compress gradient information through gradient descent. More expressive optimizers with deep memory and enhanced learning rules are proposed.
- A sequence model capable of modifying itself by learning its own update algorithm is introduced based on insights from NL on learning algorithms.
- A novel formulation for memory systems is presented, expanding on traditional long/short-term memory perspectives. The combination of the self-modifying sequence model with the continuum memory system introduces a continual learning module named Hope.
- Hope demonstrates promising results in various tasks such as language modeling, knowledge incorporation, few-shot generalization tasks, continual learning, and long-context reasoning tasks.
- Performance evaluations show that Hope outperforms attention-free models in Needle-in-a-Haystack tasks and achieves superior performance compared to linear memories and deep memory modules like Titans. In BABILong benchmark evaluation, Hope demonstrates competitive performance against large models like GPT4 and middle-size models like Llama-8B augmented with RAG.
- The study showcases how NL enhances machine learning algorithms by introducing more levels of expressiveness for higher-order in-context learning capabilities. Through detailed evaluations on various benchmarks, the effectiveness of NL in addressing fundamental challenges in deep learning architectures is highlighted.

## Summary

In this paper, the authors introduce  as a new learning paradigm that represents machine learning models with nested, multi-level, and/or parallel optimization problems. NL aims to address the challenges of continual learning, memorization, self-improvement, and finding effective solutions in deep learning models. The authors present three core contributions to support NL: 

1. The authors demonstrate that gradient-based optimizers like Adam and SGD with Momentum function as associative memory modules that compress gradient information through gradient descent. They propose more expressive optimizers with deep memory and enhanced learning rules. 2. Leveraging insights from NL on learning algorithms, the authors introduce a sequence model capable of modifying itself by learning its own update algorithm. 3. A novel formulation for memory systems is presented, expanding on traditional long/short-term memory perspectives. By combining the self-modifying sequence model with the continuum memory system, a continual learning module named Hope is introduced. Hope shows promising results in language modeling, knowledge incorporation, few-shot generalization tasks, continual learning, and long-context reasoning tasks. The authors evaluate Hope's performance in long context understanding using benchmarks such as Needle-in-a-Haystack tasks and BABILong benchmark. In Needle-in-a-Haystack tasks, outperforms attention-free models across different setups and achieves superior performance compared to linear memories and deep memory modules like Titans. In BABILong benchmark evaluation, demonstrates competitive performance against large models like GPT4 and middle-size models like Llama-8B augmented with RAG. Overall, the study showcases how  can enhance machine learning algorithms by introducing more levels of expressiveness for higher-order in-context learning capabilities. Through detailed evaluations on various benchmarks, the effectiveness of NL in addressing fundamental challenges in deep learning architectures is highlighted.