# Mass-Editing Memory in a Transformer

**Authors:** Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, David Bau  
**arXiv:** [2210.07229](https://arxiv.org/abs/2210.07229)  
**Date:** 2022-10-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.07229v1/

## Key points

- There is a growing interest in updating large language models with new memories.
- Most existing methods are limited to updating single associations.
- MEMIT is a method for directly updating a language model with many memories.
- MEMIT can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B).
- The method involves encoding each memory as a sequence of tokens and then concatenating them into one long sequence that is fed into the transformer model along with the original input text.
- The researchers conducted experiments on several benchmark datasets and found that their method significantly outperformed previous approaches in terms of accuracy and efficiency.
- They released their code and data at https://memit.baulab.info for others to use and build upon.
- MEMIT represents an important step forward in the development of language models capable of incorporating multiple memories simultaneously.

## Summary

In recent years, there has been a growing interest in updating large language models with new memories to replace outdated information or incorporate specialized knowledge. However, most of the existing methods are limited to updating single associations. In response to this limitation, a team of researchers including Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov and David Bau have developed MEMIT - a method for directly updating a language model with many memories. The team's approach demonstrates that MEMIT can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), surpassing prior work by orders of magnitude. The method involves encoding each memory as a sequence of tokens and then concatenating them into one long sequence that is fed into the transformer model along with the original input text. This allows the model to learn from both the input text and the associated memories simultaneously. The researchers conducted experiments on several benchmark datasets and found that their method significantly outperformed previous approaches in terms of accuracy and efficiency. They also released their code and data at https://memit.baulab.info for others to use and build upon. Overall, MEMIT represents an important step forward in the development of language models capable of incorporating multiple memories simultaneously. With further refinement and exploration, this approach could have significant implications for natural language processing tasks such as question answering, summarization and dialogue generation.