InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Auteurs : Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi

Soumis à arXiv le 11 Mai. 2023
Licence : CC BY 4.0
Résumé scientifique

General-purpose language models that can solve various language-domain tasks have emerged driven by the pre-training and instruction-tuning pipeline. However, building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by the additional visual input. Although vision-language pre-training has been widely studied, vision-language instruction tuning remains relatively less explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly available datasets, transform them into instruction tuning format and categorize them into two clusters for held-in instruction tuning and held-out zero-shot evaluation. Additionally, we introduce instruction-aware visual feature extraction, a crucial method that enables the model to extract informative features tailored to the given instruction. The resulting InstructBLIP models achieve state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and the larger Flamingo. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models have been open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.

Les articles autour de lui, par date et par proximité

Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article

Comment lire cette carte ? Guide illustré

Les articles les plus proches Les travaux antérieurs sur lesquels il s’appuie Les articles qui le citent

Les articles autour de celui-ci

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

  1. Très proche
    MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
    Jianhong Tu, Zhuohao Ni, Nicholas Crispino et al. · 2024 · Proceedings of the 3rd Workshop on Towards Knowledgeable Foundation Models (KnowFM) · cité 6 fois · Pas encore résumé
    Résumé scientifique

    We present a novel visual instruction tuning strategy to improve the zero-shot task generalization of multimodal large language models by building a firm text-only knowledge base. Existing work lacks sufficient experimentation on the importance of each modality in the instruction tuning stage, often using a majority of vision-language data while keeping text-only data limited and fixing mixtures of modalities. By incorporating diverse text-only data in the visual instruction tuning stage, we vary vision-language data in various controlled experiments to investigate the importance of modality in visual instruction tuning. Our comprehensive evaluation shows that the text-heavy instruction tuni…

  2. Très proche
    Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
    Myeongkyun Kang, Soopil Kim, Xiao-Xiao Li et al. · 2026 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé
    Résumé scientifique

    Large vision language models (LVLMs) have demonstrated impressive performance across a wide range of tasks. These capabilities largely stem from visual instruction tuning, which fine-tunes models on datasets consisting of curated image-instruction-output triplets. However, in the medical domain, constructing large-scale, high-quality instruction datasets is particularly challenging due to the need for specialized expert knowledge. To address this issue, we propose an instruction-free tuning approach that reduces reliance on handcrafted instructions, leveraging only image-description pairs for fine-tuning. Specifically, we introduce a momentum proxy instruction as a replacement for curated te…

  3. Très proche
    LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu et al. · 2023 · arXiv.org · preprint, non relu par les pairs · cité 335 fois · Pas encore résumé
    Résumé scientifique

    Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, e…

  4. Très proche Cité par cet article Une méthode qu’il utilise
    BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
    Junnan Li, Dongxu Li, Caiming Xiong et al. · 2022 · Computer Vision and Pattern Recognition · cité 7 646 fois · Résumé IA disponible
    Résumé scientifique

    Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We ac…

  5. Très proche Cité par cet article
    MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
    Zhiyang Xu, Ying Shen, Li-Fu Huang · 2022 · Annual Meeting of the Association for Computational Linguistics · cité 146 fois · Pas encore résumé
    Résumé scientifique

    Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multimodal tasks. In this work, we introduce MultiInstruct, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expert-written instructions. We take OFA as the base pre-trained model for multimodal instruction tuning, and to fur…

  6. Très proche
    Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
    Zhiyang Xu, Chao Feng, Rulin Shao et al. · 2024 · Annual Meeting of the Association for Computational Linguistics · cité 94 fois · Pas encore résumé
    Résumé scientifique

    Despite vision-language models' (VLMs) remarkable capabilities as versatile visual assistants, two substantial challenges persist within the existing VLM frameworks: (1) lacking task diversity in pretraining and visual instruction tuning, and (2) annotation error and bias in GPT-4 synthesized instruction tuning data. Both challenges lead to issues such as poor generalizability, hallucination, and catastrophic forgetting. To address these challenges, we construct Vision-Flan, the most diverse publicly available visual instruction tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances sourced from academic datasets, and each task is accompanied by an expert-written instru…

  7. Très proche
    MMInstruct: a high-quality multi-modal instruction tuning dataset with extensive diversity
    Yang-Zhou Liu, Yue Cao, Zhang-Wei Gao et al. · 2024 · Science China Information Sciences · cité 53 fois · Pas encore résumé
    Résumé scientifique

    Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of vision large language models (VLLMs), existing visual instruction tuning datasets include the following limitations. (1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance, instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations. (2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model’s ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visu…

  8. Très proche Cité par cet article Utilisé en profondeur par cet article Une méthode qu’il utilise
    Visual Instruction Tuning
    Haotian Liu, Chunyuan Li, Qingyang Wu et al. · 2023 · Computer Vision and Pattern Recognition · cité 11 882 fois · Résumé IA disponible
    Résumé scientifique

    Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the beha…

Les travaux antérieurs sur lesquels il s’appuie

  1. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise
    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
    Junnan Li, Dongxu Li, Silvio Savarese et al. · 2023 · Computer Vision and Pattern Recognition · cité 9 775 fois · Résumé IA disponible
    Résumé scientifique

    The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various visio…

  2. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise
    Scaling Instruction-Finetuned Language Models
    Hyung Won Chung, Le Hou, Shayne Longpre et al. · 2022 · Machine Learning · cité 4 580 fois · Résumé IA disponible
    Résumé scientifique

    Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a larg…

  3. Proche
    Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
    Or Honovich, Thomas Scialom, Omer Levy et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cité 516 fois · Pas encore résumé
    Résumé scientifique

    Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions. These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions. In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor. We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth. This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs. Experiments show tha…

  4. Proche
    12-in-1: Multi-Task Vision and Language Representation Learning
    Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach et al. · 2019 · Computer Vision and Pattern Recognition · cité 522 fois · Pas encore résumé
    Résumé scientifique

    Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly. In this work, we investigate these relationships between vision-and-language tasks by developing a large-scale, multi-task model. Our approach culminates in a single model on 12 datasets from four broad categories of task including visual question answering, caption-based image retrieval, grounding referring expressions, and multimodal verification. Compared to independently trained single-task models, this represents a reduction f…

  5. Proche Une méthode qu’il utilise
    LAVIS: A Library for Language-Vision Intelligence
    Dongxu Li, Junnan Li, Hung Le et al. · 2022 · arXiv.org · preprint, non relu par les pairs · cité 86 fois · Pas encore résumé
    Résumé scientifique

    We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications. LAVIS aims to serve as a one-stop comprehensive library that brings recent advancements in the language-vision field accessible for researchers and practitioners, as well as fertilizing future research and development. It features a unified interface to easily access state-of-the-art image-language, video-language models and common datasets. LAVIS supports training, evaluation and benchmarking on a rich variety of tasks, including multimodal classification, retrieval, captioning, visual question answering, dialogue and pre-training. In the meantime, the library is also highly extensible an…

Les articles qui le citent

  1. Proche
    Otter: A Multi-Modal Model With In-Context Instruction Tuning
    Bo Li, Yuanhan Zhang, Liangyu Chen et al. · 2025 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cité 253 fois · Pas encore résumé
    Résumé scientifique

    Recent advances in Large Multimodal Models (LMMs) have unveiled great potential as visual assistants. However, most existing works focus on responding to individual instructions or using previous dialogues for contextual understanding. There is little discussion on employing both images and text as in-context examples to enhance the instruction following capability. To bridge this gap, we introduce the Otter model to leverage both textual and visual in-context examples for instruction tuning. Specifically, Otter builds upon Flamingo with Perceiver architecture, and has been instruction tuned for general purpose multi-modal assistant. Otter seamlessly processes multi-modal inputs, supporting …

  2. Proche
    Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
    Zhe Chen, Jiannan Wu, Wenhai Wang et al. · 2024 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cité 838 fois · Pas encore résumé
    Résumé scientifique

    The exponential growth of large language models (LLMs) has opened up numerous possibilities for multi-modal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs. In this work, we design a large-scale vision-language foun-dation model (Intern VL), which scales up the vision foun-dation model to 6 billion parameters and progressively aligns it with the LLM, using web-scale image-text data from various sources. This model can be broadly applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks including visual perception tasks such as image-level o…

  3. Proche
    VILA: On Pre-training for Visual Language Models
    Ji Lin, Hongxu Yin, Wei Ping et al. · 2024 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cité 282 fois · Pas encore résumé
    Résumé scientifique

    Visual language models (VLMs) rapidly progressed with the recent success of large language models. There have been growing efforts on visual instruction tuning to extend the LLM with visual inputs, but lacks an in-depth study of the visual language pre-training process, where the model learns to perform joint modeling on both modalities. In this work, we examine the design options for VLM pre-training by augmenting LLM towards VLM through step-by-step controllable comparisons. We introduce three main findings: (1) freezing LLMs during pre-training can achieve decent zero-shot performance, but lack in-context learning capability, which requires unfreezing the LLM; (2) interleaved pre-training…

  4. Proche Utilise cet article en profondeur
    Mechanism-Level Evaluation for Vision-Language Models: Controlled Activation-Replacement Diagnosis of Gender Bias
    Zhi-Peng Zhao, Wen-Xu Wang, Pei-Shun Liu et al. · 2026 · Conference on Empirical Methods in Natural Language Processing · Pas encore résumé
    Résumé scientifique

    Behavioral benchmarking reveals \emph{what} biases exist in vision-language models but not \emph{which internal components} are most sensitive to targeted intervention, precluding principled intervention. We argue for mechanism-level evaluation as a necessary complement, demonstrating causal mediation analysis as a diagnostic instrument for gender bias. We decompose gender-cue effects into controlled indirect effects attributable to specific-layer activations and direct effects through all other pathways, producing layer-by-layer mechanistic signatures. Across six models spanning three architectural families (LLaVA-1.5, LLaVA-NeXT, InstructBLIP at 7B/13B) and two 8B-scale architectures, thre…

  5. Proche
    DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection
    Yu-Yang Hong, Jin-Hui Guo, Jia-Qi Gu et al. · 2026 · Conference on Empirical Methods in Natural Language Processing · cité 1 fois · Pas encore résumé
    Résumé scientifique

    Visual instruction tuning is crucial for advancing the vision-language alignment and instruction-following capabilities of Vision-Language Models (VLMs). However, identifying optimal subsets under a fixed ratio constraint from rapidly expanding datasets remains a significant bottleneck. While existing methods largely depend on distribution diversity or heuristic filtering, they often overlook the internal coherence within individual samples. To bridge this gap, we propose Data Intrinsic Consistency (DIC), a self-scoring metric designed to quantify the sample-level inter-component consistency. DIC consists of two modules: Visual Information Consistency (VIC), evaluating the alignment between …

Comment ces articles sont-ils trouvés ?

Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.

Mis à jour le 6 novembre 2026.

Le guide illustré des articles autour d’un article