# Unlimiformer: Long-Range Transformers with Unlimited Length Input

**Authors:** Amanda Bertsch, Uri Alon, Graham Neubig, Matthew R. Gormley  
**arXiv:** [2305.01625](https://arxiv.org/abs/2305.01625)  
**Date:** 2023-05-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.01625v1/

## Key points

- Unlimiformer is a novel approach that addresses the input length limitation of transformer-based models
- Transformer models have a predefined bound to their input length because they need to attend to every token in the input, which can be computationally expensive
- Unlimiformer proposes a general approach that can wrap any existing pretrained encoder-decoder transformer and offload the attention computation across all layers to a single $k$-nearest-neighbor index
- This index can be kept on either the GPU or CPU memory and queried in sub-linear time, allowing extremely long input sequences to be indexed without truncation at test time
- Every attention head in every decoder layer retrieves its top-$k$ keys instead of attending to every key, improving efficiency and accuracy
- The efficacy of Unlimiformer was demonstrated on several long-document and multi-document summarization benchmarks, including the BookSum dataset with inputs up to 350k tokens long
- Unlimiformer improves pretrained models such as BART and Longformer by extending them to unlimited inputs without additional learned weights or modifying their code
- The authors make their code and models publicly available at https://github.com/abertsch72/unlimiformer
- This work has significant implications for natural language processing tasks that require processing of lengthy texts, such as document summarization or machine translation. By overcoming the input length limitation of transformer-based models, Unlimiformer opens up new possibilities for more accurate and efficient processing of lengthy texts.

## Summary

Unlimiformer is a novel approach that addresses the input length limitation of transformer-based models. These models have a predefined bound to their input length because they need to attend to every token in the input, which can be computationally expensive. Unlimiformer proposes a general approach that can wrap any existing pretrained encoder-decoder transformer and offload the attention computation across all layers to a single $k$-nearest-neighbor index. This index can be kept on either the GPU or CPU memory and queried in sub-linear time, allowing extremely long input sequences to be indexed without truncation at test time. Every attention head in every decoder layer retrieves its top-$k$ keys instead of attending to every key, improving efficiency and accuracy. The efficacy of Unlimiformer was demonstrated on several long-document and multi-document summarization benchmarks, including the BookSum dataset with inputs up to 350k tokens long. Unlimiformer improves pretrained models such as BART and Longformer by extending them to unlimited inputs without additional learned weights or modifying their code. The authors make their code and models publicly available at https://github.com/abertsch72/unlimiformer. This work has significant implications for natural language processing tasks that require processing of lengthy texts, such as document summarization or machine translation. By overcoming the input length limitation of transformer-based models, Unlimiformer opens up new possibilities for more accurate and efficient processing of lengthy texts.