# XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference

**Authors:** João Monteiro, Étienne Marcotte, Pierre-André Noël, Valentina Zantedeschi, David Vázquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian  
**arXiv:** [2404.15420](https://arxiv.org/abs/2404.15420)  
**Date:** 2024-04-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.15420v1/

## Key points

- Study focuses on challenges of in-context learning (ICL) approaches and their use of prompting for decoder-only language model generation
- Introduces models inspired by encoder-decoder architecture utilizing cross-attention to condition generation on reference text without prompts
- Models outperform ICL methods, comparable to fine-tuned prompted LLMs, and reduce space footprint compared to standard KV caching
- All models experience a decline in accuracy in out-of-distribution scenarios due to variations in writing style, document lengths, signal-to-noise ratio, and distracting content
- Enhancing the robustness of context-conditional models is identified as a promising avenue for future research
- Importance of ethical considerations in developing efficient learning models is emphasized

## Summary

The study "XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference" delves into the challenges of in-context learning (ICL) approaches and their use of prompting for decoder-only language model generation. To address the inefficiency of just-in-time processing and space constraints in caching transformer states, the research introduces models inspired by the encoder-decoder architecture that utilize cross-attention to condition generation on reference text without prompts. By leveraging pre-trained decoder-only models and training a small number of additional layers, these models outperform ICL methods, are comparable to fine-tuned prompted LLMs, and significantly reduce space footprint compared to standard KV caching. However, when evaluated in an out-of-distribution scenario where test datasets differ greatly from training data, all models experience a decline in accuracy. This discrepancy may be due to variations in writing style, document lengths, signal-to-noise ratio, and distracting content related to questions but not useful for answers. Therefore, enhancing the robustness of context-conditional models is identified as a promising avenue for future research. Furthermore,of AI research is acknowledged and the importance of ethical considerations in developing efficient learning models is emphasized. The proposed architecture allows for improved performance on unseen datasets while also highlighting opportunities for further advancements in model generalization and robustness.