Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens
AI-generated Key Points
- Transformer models are foundational in NLP and computer vision
- The authors propose a VIP-token centric compression (Vcc) scheme that reduces the dependency of a Transformer model's complexity on sequence length
- Vcc selectively compresses input sequences based on their impact on approximating the representation of VIP-tokens, which are small subsets of special tokens most relevant to final predictions in many tasks
- The proposed algorithm achieves more than 3x efficiency improvement compared to baselines on 4K and 16K lengths while also achieving competitive or better performance on various tasks
- It can be scaled up to 128K tokens while consistently offering accuracy improvement
- NarrativeQA is an ideal testbed for scaling experiments since it involves longer sequences and shows that their method can be scaled to much longer sequences and achieve higher performance as sequence length increases.
- Using a few layers of standard Transformer layers to preprocess tokens helps performance and segmenting input sequence into multiple segments of 512 length before using vanilla computation in initial stages.
- For encoder-only architecture, they compare their method with RoBERTa and two strong baselines: Longformer and Big Bird.
- The proposed VIP-token centric compression scheme offers significant efficiency improvements while maintaining or improving performance on various tasks.
Authors: Zhanpeng Zeng, Cole Hawkins, Mingyi Hong, Aston Zhang, Nikolaos Pappas, Vikas Singh, Shuai Zheng
Abstract: Transformer models are foundational to natural language processing (NLP) and computer vision. Despite various recent works devoted to reducing the quadratic cost of such models (as a function of the sequence length $n$), dealing with ultra long sequences efficiently (e.g., with more than 16K tokens) remains challenging. Applications such as answering questions based on an entire book or summarizing a scientific article are inefficient or infeasible. In this paper, we propose to significantly reduce the dependency of a Transformer model's complexity on $n$, by compressing the input into a representation whose size $r$ is independent of $n$ at each layer. Specifically, by exploiting the fact that in many tasks, only a small subset of special tokens (we call VIP-tokens) are most relevant to the final prediction, we propose a VIP-token centric compression (Vcc) scheme which selectively compresses the input sequence based on their impact on approximating the representation of these VIP-tokens. Compared with competitive baselines, the proposed algorithm not only is efficient (achieving more than $3\times$ efficiency improvement compared to baselines on 4K and 16K lengths), but also achieves competitive or better performance on a large number of tasks. Further, we show that our algorithm can be scaled to 128K tokens (or more) while consistently offering accuracy improvement.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Welcome to our AI assistant! Here are some important things to keep in mind:
- The assistant will only answer questions related to this specific paper.
- Please note that this is not a bot for casual chatting.
- If you want the answer in a language other than the language you chose for navigating the website, simply add "TRANSLATE IN LANGUAGE L" at the end of your query (replace "LANGUAGE L" with the language of your choice).
- For example, you could ask "Can you extract the most important aspect of the paper? TRANSLATE IN SPANISH".
- If you want to keep the history of your questions/answers you should create an account.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Similar papers summarized with our AI tools
Navigate through even more similar papers through atree representation
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.