# In-context Autoencoder for Context Compression in a Large Language Model

**Authors:** Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, Furu Wei  
**arXiv:** [2307.06945](https://arxiv.org/abs/2307.06945)  
**Date:** 2023-07-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.06945v1/

## Key points

- The paper presents the In-Context Autoencoder (ICAE), a novel approach for context compression in large language models (LLMs)
- The ICAE consists of two modules: a learnable encoder adapted from an LLM using LoRA and a fixed decoder that is the target LLM
- The authors pretrain the ICAE using autoencoding and language modeling objectives on extensive text data
- They fine-tune the pretrained ICAE using a small amount of instructive data to enhance its interaction with different prompts and produce desirable responses
- Experimental results show that the proposed pretraining and fine-tuning paradigm enables the ICAE to produce memory slots with 4 times context compression
- These compressed memory slots can be well conditioned on by the target LLM to respond to various prompts
- The ICAE has significant implications in addressing long context problems and reducing computation and memory overheads in LLM inference
- The authors plan to release their code and data soon, contributing to further research efforts in advancing context management in large language models.

## Summary

The paper presents the In-Context Autoencoder (ICAE), a novel approach for context compression in large language models (LLMs). The ICAE consists of two modules: a learnable encoder adapted from an LLM using LoRA which compresses a long context into a limited number of memory slots, and a fixed decoder that is the target LLM capable of conditioning on these memory slots for various purposes. To train the ICAE, the authors first pretrain it using both autoencoding and language modeling objectives on extensive text data. This enables the model to generate memory slots that accurately represent the original context. Then they fine-tune the pretrained ICAE using a small amount of instructive data to enhance its interaction with different prompts and produce desirable responses. Experimental results demonstrate that the proposed pretraining and fine-tuning paradigm effectively enables the ICAE to produce memory slots with 4 times context compression. These compressed memory slots can be well conditioned on by the target LLM to respond to various prompts. The promising results highlight significant implications of the ICAE in addressing long context problems and reducing computation and memory overheads in LLM inference. The authors plan to release their code and data soon. Overall, this research contributes to advancing context management in large language models, suggesting further research efforts in this area.