# MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens

**Authors:** Yu Chen, Runkai Chen, Sheng Yi, Xinda Zhao, Xiaohong Li, Jianjin Zhang, Jun Sun, Chuanrui Hu, Yunyun Han, Lidong Bing, Yafeng Deng, Tianqiao Chen  
**arXiv:** [2603.23516](https://arxiv.org/abs/2603.23516)  
**Date:** 2026-03-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2603.23516v1/

## Key points

- Long-term memory is crucial in enhancing human intelligence in the field of artificial intelligence
- Traditional full-attention architectures have limitations on context length for large language models (LLMs)
- Various approaches like hybrid linear attention, fixed-size memory states, and external storage methods have been explored to extend context limits
- Memory Sparse Attention (MSA) model addresses challenges faced by existing methods and is efficient and massively scalable
- MSA introduces innovations like scalable sparse attention mechanisms and document-wise RoPE for linear complexity in training and inference phases
- MSA demonstrates less than 9% degradation when scaling from 16K to 100M tokens and enables efficient inference using just 2xA800 GPUs
- MSA outperforms leading LLMs, state-of-the-art RAG systems, and prominent memory agents in benchmark tests for long-context scenarios

## Summary

, , , , 

In the field of artificial intelligence, long-term memory plays a crucial role in enhancing human intelligence. The ability to process vast amounts of information over a lifetime has been a longstanding goal in AI research. However, traditional full-attention architectures have limitations, restricting the effective context length of large language models (LLMs) to around 1 million tokens. Various approaches have been explored to extend this limit, such as hybrid linear attention, fixed-size memory states like RNNs, and external storage methods like RAG or agent systems. Despite these efforts, existing methods often face challenges such as precision degradation, latency issues with growing context length, inability to dynamically modify memory content, and lack of end-to-end optimization. These bottlenecks hinder complex tasks like large-corpus summarization, Digital Twins development, and long-history agent reasoning. They also constrain memory capacity and slow down inference processes. To address these challenges, the Memory Sparse Attention (MSA) model has been introduced as an end-to-end trainable framework that is efficient and massively scalable. Key innovations within MSA include scalable sparse attention mechanisms and document-wise RoPE (Relative Positional Encoding), enabling linear complexity in both training and inference phases while maintaining stability. Remarkably, MSA demonstrates less than 9% degradation when scaling from 16K to 100M tokens. Additionally, MSA leverages KV cache compression along with Memory Parallel processing to enable efficient inference on 100M tokens using just 2xA800 GPUs. The model also introduces Memory Interleaving to support complex multi-hop reasoning across scattered memory segments. In benchmark tests for long-context scenarios, MSA outperforms leading LLMs, state-of-the-art RAG systems, and prominent memory agents. These results highlight the effectiveness of MSA in decoupling memory capacity from reasoning processes. By providing a scalable foundation for general-purpose models with intrinsic lifetime-scale memory capabilities, MSA opens up new possibilities for advanced AI applications requiring extensive memory resources and sophisticated reasoning abilities.