# InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

**Authors:** Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi  
**arXiv:** [2305.06500](https://arxiv.org/abs/2305.06500)  
**Date:** 2023-05-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.06500v1/

## Key points

- InstructBLIP is a novel instruction tuning framework for building generalized vision-language models.
- Building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by additional visual input.
- Vision-language pre-training has been widely studied, but vision-language instruction tuning remains relatively less explored.
- InstructBLIP conducts a comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models.
- The authors gather 26 publicly available datasets, transform them into instruction-tuning format and categorize them into two clusters for held-in instruction tuning and held out zero shot evaluation.
- InstructBLIP introduces an instruction aware visual feature extraction method that enables the model to extract informative features tailored to the given instructions.
- The resulting InstructBLIP models achieve state of the art zero shot performance across all 13 held out datasets substantially outperforming BLIP 2 and the larger Flamingo.
- These models lead to state of the art performance when fine tuned on individual downstream tasks such as achieving 90.7% accuracy on ScienceQA IMG.
- Qualitative examples demonstrate InstructBLIP's various capabilities in complex visual reasoning knowledge grounded image description and multi turn conversations.
- InstructBLIP can serve as an enhanced model initialization for downstream task fine tuning while achieving state of the art results.
- The paper concludes with a call for new research in general purpose multimodal AI and its applications spurred by InstructBLIP's capabilities towards building generalized vision language models.

## Summary

In this paper, the authors present InstructBLIP, a novel instruction tuning framework for building generalized vision-language models. While general-purpose language models have emerged through pre-training and instruction-tuning pipelines, building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by additional visual input. Although vision-language pre-training has been widely studied, vision-language instruction tuning remains relatively less explored. InstructBLIP addresses this gap by conducting a comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models. The authors gather 26 publicly available datasets, transform them into instruction-tuning format and categorize them into two clusters for held-in instruction tuning and held out zero shot evaluation. Moreover, InstructBLIP introduces an instruction aware visual feature extraction method that enables the model to extract informative features tailored to the given instructions. The resulting InstructBLIP models achieve state of the art zero shot performance across all 13 held out datasets substantially outperforming BLIP 2 and the larger Flamingo. Additionally these models lead to state of the art performance when fine tuned on individual downstream tasks such as achieving 90.7% accuracy on ScienceQA IMG. Furthermore qualitative examples demonstrate InstructBLIP's various capabilities in complex visual reasoning knowledge grounded image description and multi turn conversations. The authors also show that InstructBLIP can serve as an enhanced model initialization for downstream task fine tuning while achieving state of the art results. The paper concludes with a call for new research in general purpose multimodal AI and its applications spurred by InstructBLIP's capabilities towards building generalized vision language models.