# An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

**Authors:** Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, Baobao Chang  
**arXiv:** [2403.06764](https://arxiv.org/abs/2403.06764)  
**Date:** 2024-03-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.06764v1/

## Key points

- Large Vision-Language Models (LVLMs) have gained traction in computer vision and natural language processing
- FastV is a new method introduced to optimize computational efficiency within LVLMs
- FastV works by learning adaptive attention patterns in early layers and pruning visual tokens in subsequent layers
- FastV can significantly reduce computational costs without compromising performance across various image and video understanding tasks
- FastV offers customizable trade-offs between computational efficiency and performance based on specific requirements
- Practical implications of FastV include deployment on edge devices and commercial models with limited computational resources

## Summary

, , , , 


In recent years, Large Vision-Language Models (LVLMs) have gained significant traction in the fields of computer vision and natural language processing. These powerful models, such as LLaVA-1.5, QwenVL-Chat, and Video-LLaVA, have been instrumental in a wide range of applications, from image description to internet navigation and decision-making in real-world scenarios. However, a key challenge identified in these LVLMs is the inefficient attention computation over visual tokens in the deep layers, leading to computational inefficiencies compared to handling textual data. To address this issue, a new method called FastV has been introduced as a versatile plug-and-play solution aimed at optimizing computational efficiency within LVLMs. FastV works by learning adaptive attention patterns in early layers and pruning visual tokens in subsequent layers. Through extensive evaluations, it has been demonstrated that FastV can significantly reduce computational costs (e.g., achieving a 45% reduction in FLOPs for LLaVA-1.5-13B) without compromising performance across various image and video understanding tasks. One of the key strengths of FastV lies in its customizable nature, allowing users to tailor the trade-off between computational efficiency and performance based on their specific requirements. Remarkably, FastV can compress the FLOPs of a 13B-parameter model to achieve better efficiency than that of a 7B-parameter model while maintaining superior performance levels. The practical implications of FastV are substantial, particularly for deployment on edge devices and commercial models where computational resources are limited. By unlocking the potential of LVLMs through efficient attention mechanisms and token pruning strategies, FastV offers a promising pathway towards enhancing the scalability and applicability of large-scale vision-language models in real-world settings. The code for implementing FastV is openly available at https://github.com/pkunlp-icler/FastV. In conclusion, the introduction of FastV represents a significant advancement in addressing the inefficiencies observed in current LVLMs. By offering a scalable and efficient solution for optimizing attention mechanisms within these models, FastV opens up new possibilities for leveraging LVLMs across diverse applications with improved computational efficiency and performance outcomes.