LVLM-Intrepret: An Interpretability Tool for Large Vision-Language Models

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close* Not relevant

   LVLM-Intrepret: An Interpretability Tool for Large Vision-Language Models

   Gabriela Ben Melech Stan, R. Y. Rohekar, Yaniv Gurwicz et al. · 2024 · XAI4CV · cited 47 times · Not summarized yet
2. Very close Not relevant

   An Introduction to Vision-Language Modeling

   Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay et al. · 2024 · Machine Learning · preprint, not peer reviewed · cited 270 times · AI summary available

   Abstract

   Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision …

3. Very close Not relevant

   Patchwise Cooperative Game-based Interpretability Method for Large Vision-language Models

   Yao Zhu, Yun-Jian Zhang, Zi-Zhe Wang et al. · 2025 · Transactions of the Association for Computational Linguistics · cited 4 times · Not summarized yet

   Abstract

   Amidst the rapid advancement of artificial intelligence, research on large vision-language models (LVLMs) has emerged as a pivotal area. However, understanding their internal mechanisms remains challenging due to the limitations of existing interpretability methods, especially regarding faithfulness and plausibility. To address this, we first construct a human response interpretability dataset that evaluates the plausibility of model explanations by comparing the attention regions between the model and humans when answering the same questions. We then propose a patchwise cooperative game-based interpretability method for LVLMs, which employs Shapley values to quantify the impact of individua…

4. Very close Not relevant

   Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering

   Zeping Yu, Sophia Ananiadou · 2024 · arXiv.org · preprint, not peer reviewed · cited 13 times · Not summarized yet

   Abstract

   Understanding the mechanisms behind Large Language Models (LLMs) is crucial for designing improved models and strategies. While recent studies have yielded valuable insights into the mechanisms of textual LLMs, the mechanisms of Multi-modal Large Language Models (MLLMs) remain underexplored. In this paper, we apply mechanistic interpretability methods to analyze the visual question answering (VQA) mechanisms in the first MLLM, Llava. We compare the mechanisms between VQA and textual QA (TQA) in color answering tasks and find that: a) VQA exhibits a mechanism similar to the in-context learning mechanism observed in TQA; b) the visual features exhibit significant interpretability when projecti…

5. Very close Not relevant

   Language Model as Visual Explainer

   Xingyi Yang, Xin-Chao Wang · 2024 · Neural Information Processing Systems · cited 5 times · Not summarized yet

   Abstract

   In this paper, we present Language Model as Visual Explainer LVX, a systematic approach for interpreting the internal workings of vision models using a tree-structured linguistic explanation, without the need for model training. Central to our strategy is the collaboration between vision models and LLM to craft explanations. On one hand, the LLM is harnessed to delineate hierarchical visual attributes, while concurrently, a text-to-image API retrieves images that are most aligned with these textual concepts. By mapping the collected texts and images to the vision model's embedding space, we construct a hierarchy-structured visual embedding tree. This tree is dynamically pruned and grown by q…

6. Very close Not relevant

   Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability

   Dong Shu, Hai-Yan Zhao, Jing-Yu Hu et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 37 times · Not summarized yet

   Abstract

   Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in processing both visual and textual information. However, the critical challenge of alignment between visual and textual representations is not fully understood. This survey presents a comprehensive examination of alignment and misalignment in LVLMs through an explainability lens. We first examine the fundamentals of alignment, exploring its representational and behavioral aspects, training methodologies, and theoretical foundations. We then analyze misalignment phenomena across three semantic levels: object, attribute, and relational misalignment. Our investigation reveals that misalignment emerges from challen…

7. Very close Cites this paper Not relevant

   GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Models

   Guanxin Shen · 2025 · cited 3 times · Not summarized yet

   Abstract

   Recent large vision-language models (LVLMs) have advanced capabilities in visual question answering (VQA). However, interpreting where LVLMs direct their visual attention remains a significant challenge, yet is essential for understanding model behavior. We introduce GLIMPSE (Gradient-Layer Importance Mapping for Prompted Visual Saliency Explanation), a lightweight, model-agnostic framework that jointly attributes LVLM outputs to the most relevant visual evidence and textual signals that support open-ended generation. GLIMPSE fuses gradient-weighted attention, adaptive layer propagation, and relevance-weighted token aggregation to produce holistic response-level heat maps for interpreting cr…

8. Very close Not relevant

   Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey

   Yunkai Dang, Kaichen Huang, Jiahao Huo et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 94 times · Not summarized yet

   Abstract

   The rapid development of Artificial Intelligence (AI) has revolutionized numerous fields, with large language models (LLMs) and computer vision (CV) systems driving advancements in natural language understanding and visual processing, respectively. The convergence of these technologies has catalyzed the rise of multimodal AI, enabling richer, cross-modal understanding that spans text, vision, audio, and video modalities. Multimodal large language models (MLLMs), in particular, have emerged as a powerful framework, demonstrating impressive capabilities in tasks like image-text generation, visual question answering, and cross-modal retrieval. Despite these advancements, the complexity and scal…

Prior work it builds on

1. Very close A method it uses Not relevant

   Visual Instruction Tuning

   Haotian Liu, Chunyuan Li, Qingyang Wu et al. · 2023 · Computer Vision and Pattern Recognition · cited 11,807 times · AI summary available

   Abstract

   Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the beha…

2. Close Not relevant

   Analyzing and Mitigating Object Hallucination in Large Vision-Language Models

   Yi-Yang Zhou, Chen-Hang Cui, Jaehong Yoon et al. · 2023 · International Conference on Learning Representations · cited 399 times · Not summarized yet

   Abstract

   Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages. However, LVLMs still suffer from object hallucination, which is the problem of generating descriptions that include objects that do not actually exist in the images. This can negatively impact many vision-language tasks, such as visual summarization and reasoning. To address this issue, we propose a simple yet powerful algorithm, LVLM Hallucination Revisor (LURE), to post-hoc rectify object hallucination in LVLMs by reconstructing less hallucinatory descriptions. LURE is grounded in a rigorous statistical analysis of the key factors underlying object hallucination, i…

3. Close* A method it uses Not relevant

   Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities

   Jinze Bai, Shuai Bai, Shu-Sheng Yang et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 2,000 times · Not summarized yet
4. Close Not relevant

   Vision-Language Transformer for Interpretable Pathology Visual Question Answering

   Usman Naseem, M. Khushi, Jinman Kim · 2022 · IEEE journal of biomedical and health informatics · cited 69 times · Not summarized yet

   Abstract

   Pathology visual question answering (PathVQA) attempts to answer a medical question posed by pathology images. Despite its great potential in healthcare, it is not widely adopted because it requires interactions on both the image (vision) and question (language) to generate an answer. Existing methods focused on treating vision and language features independently, which were unable to capture the high and low-level interactions that are required for VQA. Further, these methods failed to offer capabilities to interpret the retrieved answers, which are obscure to humans where the models’ interpretability to justify the retrieved answers has remained largely unexplored. Motivated by these limit…

5. Close Not relevant

   Interpretability-Aware Vision Transformer

   Yao Qiang, Chengyin Li, Hui Zhu et al. · 2023 · IEEE International Joint Conference on Neural Network · cited 14 times · Not summarized yet

   Abstract

   Vision Transformers (ViTs) have become prominent models for solving various vision tasks. However, the interpretability of ViTs has not kept pace with their promising performance. While there has been a surge of interest in developing post hoc solutions to explain ViTs’ outputs, these methods do not generalize to different downstream tasks and various transformer architectures. Furthermore, if ViTs are not properly trained with the given data and do not prioritize the region of interest, the post hoc methods become less effective. To overcome this limitation, we introduce a novel training procedure that inherently enhances ViT’s interpretability. Our interpretability-aware ViT (IA-ViT) draws…

Papers that cite it

1. Close Uses this paper in depth Not relevant

   FastRM: An efficient and automatic explainability framework for multimodal generative models

   Gabriela Ben Melech Stan, E. Aflalo, Man Luo et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Abstract

   Large Vision Language Models (LVLMs) have demonstrated remarkable reasoning capabilities over textual and visual inputs. However, these models remain prone to generating misinformation. Identifying and mitigating ungrounded responses is crucial for developing trustworthy AI. Traditional explainability methods such as gradient-based relevancy maps, offer insight into the decision process of models, but are often computationally expensive and unsuitable for real-time output validation. In this work, we introduce FastRM, an efficient method for predicting explainable Relevancy Maps of LVLMs. Furthermore, FastRM provides both quantitative and qualitative assessment of model confidence. Experimen…

2. Close Uses this paper in depth Not relevant

   Towards Understanding Multimodal Fine-Tuning: Spatial Features

   Lachin Naghashyar, Hunar Batra, Ashkan Khakzar et al. · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   Contemporary Vision-Language Models (VLMs) achieve strong performance on a wide range of tasks by pairing a vision encoder with a pre-trained language model, fine-tuned for visual-text inputs. Yet despite these gains, it remains unclear how language backbone representations adapt during multimodal training and when vision-specific capabilities emerge. In this work, we present the first mechanistic analysis of VLM adaptation. Using stage-wise model diffing, a technique that isolates representational changes introduced during multimodal fine-tuning, we reveal how a language model learns to"see". We first identify vision-preferring features that emerge or reorient during fine-tuning. We then sh…

3. Very close Not relevant

   Augmented Vision-Language Models: A Systematic Review

   Anthony C Davis, Burhan A. Sadiq, Tian-Min Shu et al. · 2025 · Trans. Mach. Learn. Res. · Not summarized yet

   Abstract

   Recent advances in visual-language machine learning models have demonstrated exceptional ability to use natural language and understand visual scenes by training on large, unstructured datasets. However, this training paradigm cannot produce interpretable explanations for its outputs, requires retraining to integrate new information, is highly resource-intensive, and struggles with certain forms of logical reasoning. One promising solution involves integrating neural networks with external symbolic information systems, forming neural symbolic systems that can enhance reasoning and memory abilities. These neural symbolic systems provide more interpretable explanations to their outputs and the…

4. Close Not relevant

   V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models

   Qi-Dong Wang, Junjie Hu, Ming Jiang · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 9 times · Not summarized yet

   Abstract

   Recent advances in causal interpretability have extended from language models to vision-language models (VLMs), seeking to reveal their internal mechanisms through input interventions. While textual interventions often target semantics, visual interventions typically rely on coarse pixel-level perturbations, limiting semantic insights on multimodal integration. In this study, we introduce V-SEAM, a novel framework that combines Visual Semantic Editing and Attention Modulating for causal interpretation of VLMs. V-SEAM enables concept-level visual manipulations and identifies attention heads with positive or negative contributions to predictions across three semantic levels: objects, attribute…

5. Close Not relevant

   MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs

   Huiyi Chen, Jia-Wei Peng, De-Hai Min et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 23 times · Not summarized yet

   Abstract

   Evaluating the robustness of Large Vision-Language Models (LVLMs) is essential for their continued development and responsible deployment in real-world applications. However, existing robustness benchmarks typically focus on hallucination or misleading textual inputs, while largely overlooking the equally critical challenge posed by misleading visual inputs in assessing visual understanding. To fill this important gap, we introduce MVI-Bench, the first comprehensive benchmark specially designed for evaluating how Misleading Visual Inputs undermine the robustness of LVLMs. Grounded in fundamental visual primitives, the design of MVI-Bench centers on three hierarchical levels of misleading vis…