# Augmenting Language Models with Long-Term Memory

**Authors:** Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei  
**arXiv:** [2306.07174](https://arxiv.org/abs/2306.07174)  
**Date:** 2023-06-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.07174v1/

## Key points

- The paper introduces a framework called Language Models Augmented with Long-Term Memory (LongMem)
- LongMem addresses the limitation of existing large language models (LLMs) in utilizing long-context information
- LongMem uses a decoupled network architecture with a memory encoder and an adaptive residual side-network as a memory retriever and reader
- LongMem allows for caching and updating long-term past contexts without suffering from memory staleness
- The proposed framework is enhanced with memory-augmented adaptation training, enabling it to memorize long past context and utilize long-term memory for language modeling
- The memory retrieval module in LongMem can handle unlimited-length context in its memory bank
- LongMem can expand the long-form memory up to 65k tokens, allowing it to cache numerous extra demonstration examples for in-context learning
- Experimental results show that LongMem outperforms strong long-context models on ChapterBreak benchmark
- LongMem achieves remarkable improvements in memory-augmented in-context learning compared to traditional LLMs
- The authors provide an open source implementation of their code at https://aka.ms/LongMem.

## Summary

The paper titled "Augmenting Language Models with Long-Term Memory" introduces a framework called Language Models Augmented with Long-Term Memory (LongMem) to address the limitation of existing large language models (LLMs) in utilizing long-context information. LLMs can only handle fixed-sized inputs, which restricts their ability to incorporate rich historical context. To overcome this, the authors propose a decoupled network architecture where the original LLM serves as a memory encoder and an adaptive residual side-network acts as a memory retriever and reader. This design allows for caching and updating long-term past contexts without suffering from memory staleness. The proposed LongMem framework is enhanced with memory-augmented adaptation training, enabling it to memorize long past context and utilize long-term memory for language modeling. The memory retrieval module in LongMem can handle unlimited-length context in its memory bank, benefiting various downstream tasks. Specifically, LongMem can expand the long-form memory up to 65k tokens, allowing it to cache numerous extra demonstration examples for in-context learning. Experimental results demonstrate that the proposed method outperforms strong long-context models on ChapterBreak, a challenging benchmark for long-context modeling. Additionally, LongMem achieves remarkable improvements in memory-augmented in-context learning compared to traditional LLMs. These findings highlight the effectiveness of the proposed method in helping language models memorize and utilize long form contents. The authors provide an open source implementation of their code at https://aka.ms/LongMem.