# M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

**Authors:** Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, Lidong Bing  
**arXiv:** [2306.05179](https://arxiv.org/abs/2306.05179)  
**Date:** 2023-06-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.05179v1/

## Key points

- M3Exam is a benchmark for evaluating large language models (LLMs) in the context of general intelligence.
- Human exams are considered more suitable for evaluating LLMs as they require a wider range of abilities.
- M3Exam dataset consists of 12,317 questions in 9 diverse languages with three educational levels.
- Approximately 23% of the questions require processing images for successful solving.
- The dataset includes context information, main question text, candidate options, correct answers, and meta information such as language, level, subject, and image requirements.
- The selected languages in M3Exam range from high-resource to extremely low-resource languages.
- Various top-performing LLMs in multilingual or multimodal settings are evaluated on M3Exam.
- Current models struggle with multilingual text, especially in low resource and non-Latin script languages.
- Multimodal LLMs perform poorly with complex multimodal questions.
- M3Exam addresses the limitations of task-specific benchmarks by incorporating multilingualism, multimodality, and a multilevel structure.

## Summary

This study introduces M3Exam, a novel benchmark for evaluating large language models (LLMs) in the context of general intelligence. The researchers argue that human exams are a more suitable means of evaluating LLMs as they require a wider range of abilities such as language understanding, domain knowledge, and problem-solving skills. M3Exam is sourced from real and official human exam questions and exhibits three unique characteristics: multilingualism, multimodality, and a multilevel structure. The M3Exam dataset consists of 12,317 questions in 9 diverse languages with three educational levels. Approximately 23% of the questions require processing images for successful solving. The dataset includes context information, main question text, candidate options, correct answers, and meta information such as language, level, subject and image requirements. The selected languages in M3Exam span a wide range from high-resource languages like English and Chinese to extremely low-resource languages like Javanese. This diversity allows for comprehensive assessment of the multilingual capabilities of LLMs. The ratio of questions requiring images varies across countries. For the experiment setups various top-performing LLMs in either multilingual or multimodal settings are selected including ChatGPT (gpt-3.5-turbo), GPT-4 (gpt-4), Claude (Claude-instant), BLOOM (176B) and Vicuna (13B). Some models are open source while others are obtained via API calls. The performance of these LLMs on M3Exam is evaluated which reveals that current models still struggle with multilingual text particularly in low resource and non Latin script languages while multimodal LLMs also perform poorly with complex multimodal questions. Overall this study presents M3Exam as a benchmark for evaluating LLMs' general intelligence addressing the limitations of task specific benchmarks by incorporating multilingualism multimodality and a multilevel structure. The data and evaluation code are available on GitHub making it possible to track the development of LLMs over time using this valuable resource for comprehensively assessing their capabilities.