# RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance

**Authors:** Matin Mortaheb, Mohammad A. Amir Khojastepour, Srimat T. Chakradhar, Sennur Ulukus  
**arXiv:** [2501.03995](https://arxiv.org/abs/2501.03995)  
**Date:** 2025-01-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.03995v1/

## Key points

- Authors Matin Mortaheb, Mohammad A. Amir Khojastepour, Srimat T. Chakradhar, and Sennur Ulukus explore retrieval-augmented generation (RAG) and its impact on large language models (LLMs).
- RAG leverages external knowledge to enhance response generation in LLMs and reduce hallucinations.
- Concerns are raised about potential new sources of hallucination in multi-modal settings with RAG.
- The authors propose a framework for evaluating multi-modal RAG using relevancy score (RS) and correctness score (CS).
- Mortaheb et al. train RS and CS models with data from ChatGPT and human evaluator samples, achieving an accuracy rate of approximately 88% on test data.
- They create a human-annotated database to evaluate relevancy of retrieved pieces and accuracy of response statements, showing high alignment with human preferences for both RS and CS models.
- The study highlights the importance of rigorously evaluating multi-modal RAG systems to improve performance in natural language processing applications.

## Summary

In their paper titled "RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance," authors Matin Mortaheb, Mohammad A. Amir Khojastepour, Srimat T. Chakradhar, and Sennur Ulukus delve into the realm of retrieval-augmented generation (RAG) and its impact on large language models (LLMs). RAG is a technique that leverages external knowledge to enhance response generation in LLMs, ultimately reducing the occurrence of hallucinations. The authors highlight a potential drawback of RAG in multi-modal settings where new sources of hallucination can emerge. They identify concerns such as the retrieval process selecting irrelevant pieces from the database and images being processed into text-based context or directly utilized by multi-modal language models like GPT-4o. To address these challenges, they propose a novel framework for evaluating multi-modal RAG through two key performance measures: relevancy score (RS) and correctness score (CS). To validate their framework, Mortaheb et al. train RS and CS models using a database derived from ChatGPT and human evaluator samples. The results show an impressive accuracy rate of approximately 88% on test data for both models. Additionally, they construct a comprehensive 5000-sample human-annotated database to evaluate the relevancy of retrieved pieces and accuracy of response statements. Their RS model aligns with human preferences more frequently than CLIP in retrieval tasks while their CS model successfully matches human preferences around 91% of the time. Finally, Mortaheb and colleagues assess various RAG systems' selection and generation performances using RS and CS metrics to provide valuable insights into improving multi-modal RAG systems effectively. This study emphasizes the importance of rigorously evaluating multi-modal retrieval-augmented generation techniques to mitigate potential sources of hallucination and enhance overall system performance in natural language processing applications.