# MoVA: Adapting Mixture of Vision Experts to Multimodal Context

**Authors:** Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, Yu Liu  
**arXiv:** [2404.13046](https://arxiv.org/abs/2404.13046)  
**Date:** 2024-04-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.13046v1/

## Key points

- Visual encoders play a crucial role in multimodal large language models (MLLMs)
- MoVA is a novel MLLM that dynamically routes and fuses task-specific vision experts using a coarse-to-fine mechanism
- Context-aware expert routing strategy selects suitable vision experts based on user instructions, input images, and expertise in the coarse-grained stage
- Introduction of mixture-of-vision-expert adapter (MoV-Adapter) in the fine-grained stage to extract and fuse task-specific knowledge from different experts
- MoVA enhances generalization ability by combining representations from experts based on multimodal context and model expertise
- Extensive experiments show significant performance gains over current state-of-the-art methods without added complexities
- Codes and models for MoVA are available at https://github.com/TempleX98/MoVA

## Summary

The paper "MoVA: Adapting Mixture of Vision Experts to Multimodal Context" by Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu delves into the crucial role of visual encoders in multimodal large language models (MLLMs) and how their performance impacts the model's understanding of diverse image content. The authors propose MoVA as a novel MLLM that dynamically routes and fuses task-specific vision experts using a coarse-to-fine mechanism. In the coarse-grained stage of MoVA, a context-aware expert routing strategy is designed to select the most suitable vision experts based on user instructions, input images, and expertise. This leverages the powerful function understanding capabilities of large language models equipped with expert-routing low-rank adaptation (LoRA). In the fine-grained stage, the authors introduce the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from different experts. This approach effectively combines representations from experts based on multimodal context and model expertise to enhance generalization ability. Extensive experiments were conducted to evaluate MoVA's effectiveness across challenging multimodal benchmarks. The results demonstrate significant performance gains over current state-of-the-art methods without any additional complexities. The codes and models for MoVA are available at https://github.com/TempleX98/MoVA. Overall, MoVA presents a promising solution for improving image content understanding in multimodal large language models through adaptive routing and fusion of task-specific vision experts.