# M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

**Authors:** Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, Mohit Bansal  
**arXiv:** [2411.04952](https://arxiv.org/abs/2411.04952)  
**Date:** 2024-11-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.04952v1/

## Key points

- Authors introduce M3DocRAG framework for Document Visual Question Answering (DocVQA) pipelines
- Existing methods focus on single-page documents, face challenges with multi-page or multi-document scenarios and visual elements
- M3DocRAG is a versatile multi-modal RAG framework for various document contexts, question complexities, and evidence modalities
- Empirical results show M3DocRAG outperforms strong baselines in MP-DocVQA when paired with specific models
- Comprehensive analyses validate effectiveness of M3DocRAG approach in handling diverse scenarios and enhancing multi-page multi-document understanding

## Summary

, , , , 

In their paper titled "M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding," authors Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal introduce a novel framework for Document Visual Question Answering (DocVQA) pipelines. The existing methods in this field primarily focus on handling single-page documents using multi-modal language models (MLMs) or rely on text-based retrieval-augmented generation (RAG) with tools like optical character recognition (OCR). However, these approaches face challenges when dealing with real-world scenarios where questions require information from multiple pages or documents and where important data may be present in visual elements such as figures. To address these limitations, the authors propose M3DocRAG, a versatile multi-modal RAG framework that can adapt to various document contexts (closed-domain and open-domain), question complexities (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.). By leveraging a combination of a multi-modal retriever and an MLM, M3DocRAG efficiently retrieves relevant documents and provides answers while preserving visual information. Additionally, the authors introduce M3DocVQA as a new benchmark for evaluating open-domain DocVQA across over 3,000 PDF documents comprising 40,000+ pages. Empirical results from three benchmarks - M3DocVQA/MMLongBench-Doc/MP-DocVQA - demonstrate that M3DocRAG outperforms several strong baselines including achieving state-of-the-art performance in MP-DocVQA when paired with ColPali and Qwen2-VL 7B models. The authors conduct comprehensive analyses of different indexing techniques, MLMs, and retrieval models to further validate their approach's effectiveness. Furthermore, qualitative assessments showcase M3DocRAG's ability to handle diverse scenarios such as extracting information across multiple pages or retrieving answer evidence solely from images. Overall, the proposed M3DocRAG framework presents a promising solution for enhancing multi-page multi-document understanding in DocVQA applications by effectively integrating visual elements into the question-answering process.