# LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

**Authors:** Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu  
**arXiv:** [2410.10813](https://arxiv.org/abs/2410.10813)  
**Date:** 2024-10-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.10813v1/

## Key points

- Recent advancements in LLM-driven chat assistant systems have integrated memory components for more accurate and personalized responses.
- LongMemEval is a benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
- Commercial chat assistants and long-context LLMs showed a 30% accuracy drop in memorizing information across sustained interactions when tested on LongMemEval.
- A unified framework was proposed to optimize long-term memory design through key design choices across indexing, retrieval, and reading stages.
- Memory design optimizations such as session decomposition, fact-augmented key expansion, and time-aware query expansion significantly improved memory recall and downstream question answering on LongMemEval.

## Summary

, , , , 

Recent advancements in large language model (LLM)-driven chat assistant systems have led to the integration of memory components, resulting in more accurate and personalized responses. However, the long-term memory capabilities of these systems during sustained interactions have not been fully explored. To address this gap, LongMemEval was introduced as a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. LongMemEval presents a significant challenge to existing long-term memory systems with 500 meticulously curated questions embedded within freely scalable user-assistant chat histories. Commercial chat assistants and long-context LLMs showed a 30% accuracy drop in memorizing information across sustained interactions when tested on LongMemEval. To optimize long-term memory design, a unified framework was proposed by breaking it down into four key design choices across indexing, retrieval, and reading stages. Experimental insights led to the proposal of several memory designs including session decomposition for optimizing value granularity, fact-augmented key expansion for enhancing the index structure, and time-aware query expansion for refining the search scope. These optimizations significantly improved both memory recall and downstream question answering on LongMemEval. In conclusion, this study highlights the need for more sophisticated memory mechanisms in LLM-based chat assistants to achieve personalized and reliable conversational AI. LongMemEval serves as a valuable benchmark to drive future advancements in long-term memory capabilities for chat assistants. The reproducibility statement emphasizes transparency in sharing resources and code to enable other researchers to replicate results and build upon findings for further advancements in the field.