# $\text{Memory}^3$: Language Modeling with Explicit Memory

**Authors:** Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, Chenyang Xi, Yu Yu, Kai Chen, Feiyu Xiong, Linpeng Tang, Weinan E  
**arXiv:** [2407.01178](https://arxiv.org/abs/2407.01178)  
**Date:** 2024-07-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.01178v1/

## Key points

- Authors introduce a groundbreaking approach to enhancing large language models (LLMs) by incorporating explicit memory
- Named $\text{Memory}^3$ to signify explicit memory as the third form of memory in LLMs after implicit memory and working memory
- Equipping LLMs with explicit memory is proposed as a cost-effective alternative to model parameters and text retrieval-augmented generation (RAG)
- Introduction of pioneering techniques such as a memory sparsification mechanism and a two-stage pretraining scheme for effective memory formation
- Research represents a significant advancement in language modeling by reducing computational costs associated with large-scale models

## Summary

In their paper titled "$\text{Memory}^3$: Language Modeling with Explicit Memory," authors Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, Chenyang Xi, Yu Yu, Kai Chen,
Feiyu Xiong,Linpeng Tang,and Weinan E introduce a groundbreaking approach to enhancing large language models (LLMs) by incorporating explicit memory. The concept behind this innovation lies in externalizing a significant portion of the LLM's knowledge into explicit memories. To demonstrate the efficacy of their approach,the authors undertake the task of training a 2.4B LLM from scratch. Named $\text{Memory}^3$ to signify explicit memory as the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values), this innovative model is supported by a novel memory circuitry theory that facilitates knowledge externalization. has long been recognized as a crucial component in natural language processing tasks. However, has not been widely explored as an avenue for improving . In their research, propose equipping LLMs with explicit memory as a more cost-effective alternative to model parameters and text retrieval-augmented generation (RAG). This not only reduces computational costs but also improves performance metrics. The training and inference processes of LLMs are traditionally resource-intensive as they involve transferring knowledge from raw data to meaningful computation. Drawing inspiration from the  of the human brain,the authors introduce pioneering techniques such as a memory sparsification mechanism for manageable storage and a two-stage pretraining scheme that aids in effective memory formation. These advancements pave the way for future developments in optimizing LLM architectures for enhanced efficiency and effectiveness. Overall, this research represents a significant advancement in language modeling by introducing explicit memory as an efficient means of reducing computational costs associated with large-scale models. The findings not only showcase improved performance metrics but also open doors for further exploration of  and its impact on .