# MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

**Authors:** Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Rongrong Ji  
**arXiv:** [2306.13394](https://arxiv.org/abs/2306.13394)  
**Date:** 2023-06-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.13394v2/

## Key points

- The Multimodal Large Language Model (MLLM) has impressive performance in multimodal tasks
- Previous case studies have not provided a comprehensive evaluation of MLLM's capabilities
- The authors introduce the first MLLM Evaluation benchmark called MME
- MME assesses perception and cognition abilities across 14 subtasks
- Instruction-answer pairs are manually designed to avoid data leakage from public datasets
- Concise instruction design enables fair comparisons between different MLLMs without relying heavily on prompt engineering
- Quantitative statistics can be easily obtained using these instructions
- 12 advanced MLLMs were evaluated on the MME benchmark
- Areas for improvement in existing models were highlighted
- Potential directions for future model optimization were suggested

## Summary

The Multimodal Large Language Model (MLLM) has gained attention for its impressive performance in multimodal tasks, such as generating poems based on images. However, previous case studies have not provided a comprehensive evaluation of MLLM's capabilities. In this paper, the authors address this gap by introducing the first MLLM Evaluation benchmark called MME. This benchmark assesses both perception and cognition abilities across 14 subtasks. To ensure unbiased evaluation, the instruction-answer pairs are manually designed to avoid data leakage from public datasets. The concise instruction design enables fair comparisons between different MLLMs without relying heavily on prompt engineering. Additionally, the authors demonstrate that quantitative statistics can be easily obtained using these instructions. The study evaluates 12 advanced MLLMs on the MME benchmark and highlights areas for improvement in existing models while suggesting potential directions for future model optimization. Overall, this research provides a valuable contribution to evaluating and advancing multimodal large language models.