# LLaVA-Critic: Learning to Evaluate Multimodal Models

**Authors:** Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, Chunyuan Li  
**arXiv:** [2410.02712](https://arxiv.org/abs/2410.02712)  
**Date:** 2024-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.02712v1/

## Key points

- Authors introduce LLaVA-Critic as the first open-source large multimodal model (LMM) designed for versatile evaluation across various tasks.
- Model trained using high-quality critic instruction-following dataset covering a wide range of evaluation criteria and scenarios.
- LMM-as-a-Judge: Provides reliable evaluation scores comparable to or surpassing GPT models on multiple benchmarks.
- Preference Learning: Generates reward signals for preference learning, enhancing model alignment capabilities.
- Significance of open-source LMMs emphasized in facilitating self-critique and evaluation processes.
- Potential of LLaVA-Critic demonstrated in providing valuable feedback mechanisms for large multimodal models, paving the way for scalable and superhuman alignment feedback mechanisms.
- Research contributes to advancing multimodal model evaluation field and highlights importance of leveraging open-source resources for improving model performance and alignment capabilities.

## Summary

In their paper titled "LLaVA-Critic: Learning to Evaluate Multimodal Models," authors Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li introduce LLaVA-Critic as the first open-source large multimodal model (LMM) designed to serve as a versatile evaluator across various multimodal tasks. The model is trained using a high-quality critic instruction-following dataset that encompasses a wide range of evaluation criteria and scenarios. Through their experiments, the authors showcase LLaVA-Critic's efficacy in two crucial areas: firstly, as an LMM-as-a-Judge where it delivers dependable evaluation scores comparable to or even surpassing GPT models on multiple evaluation benchmarks; and secondly, in Preference Learning where it generates reward signals for preference learning thereby enhancing model alignment capabilities. The study highlights the significance of open-source LMMs in facilitating self-critique and evaluation processes. By demonstrating the potential of LLaVA-Critic in providing valuable feedback mechanisms for large multimodal models, the research paves the way for future investigations into scalable and superhuman alignment feedback mechanisms for LMMs. This work not only contributes to advancing the field of multimodal model evaluation but also underscores the importance of leveraging open-source resources for enhancing model performance and alignment capabilities.