# CAMEL-Bench: A Comprehensive Arabic LMM Benchmark

**Authors:** Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Alharthi, Ines Riahi, Abduljalil Saif, Jorma Laaksonen, Fahad S. Khan, Salman Khan, Rao M. Anwer  
**arXiv:** [2410.18976](https://arxiv.org/abs/2410.18976)  
**Date:** 2024-10-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.18976v1/

## Key points

- Growing interest in developing large multimodal models (LMMs) for visual reasoning and understanding tasks
- Introduction of benchmarks for evaluating LMMs, primarily focused on English-centric evaluations
- Need for comprehensive evaluation benchmarks tailored to the Arabic language due to over 400 million Arabic speakers worldwide
- Development of CAMEL-Bench specifically for evaluating Arabic LMMs across eight diverse domains
- CAMEL-Bench includes 38 sub-domains with over 29,000 carefully curated questions by native Arabic speakers
- Data filtering and verification process ensures QA text is originally in Arabic or accurately translated from English
- Evaluation framework includes specialized metrics to assess performance of closed-source models like GPT-4 series and open-source LMMs
- Analysis shows areas for improvement among best open-source models with an overall score of 62%
- <Organization> aims to facilitate advancements in Arabic language modeling and promote inclusivity in multimodal model evaluations across different languages

## Summary

, , , , 


In recent years, there has been a growing interest in developing large multimodal models (LMMs) capable of performing various visual reasoning and understanding tasks. To evaluate the performance of these LMMs, multiple benchmarks have been introduced, primarily focusing on English-centric evaluations. However, with over 400 million Arabic speakers worldwide, there is a need for comprehensive evaluation benchmarks tailored to the Arabic language. In response to this need, a new benchmark called CAMEL-Bench has been developed specifically for evaluating Arabic LMMs. This benchmark covers eight diverse domains including multimodal understanding and reasoning, OCR and document understanding, charts and diagrams, videos, cultural-specific content, medical images, agricultural images, and remote sensing understanding in Arabic. Within these domains, CAMEL-Bench encompasses 38 sub-domains with over 29,000 carefully curated questions by native Arabic speakers. The data filtering and verification process for CAMEL-Bench involves ensuring that the QA text is originally in Arabic or accurately translated from English. Subsets of data undergo manual verification to maintain quality standards. Through this rigorous process, 29,036 high-quality questions are obtained for evaluation. The evaluation framework for CAMEL-Bench includes specialized metrics designed to assess the performance of both closed-source models like GPT-4 series and open-source LMMs. The analysis reveals areas for improvement among the best open-source models in achieving an overall score of 62%. By providing an open-sourced benchmark and evaluation scripts,<Organization> aims to facilitate advancements in Arabic language modeling and promote inclusivity in multimodal model evaluations across different languages.