# Analyzing the Efficacy of an LLM-Only Approach for Image-based Document Question Answering

**Authors:** Nidhi Hegde, Sujoy Paul, Gagan Madan, Gaurav Aggarwal  
**arXiv:** [2309.14389](https://arxiv.org/abs/2309.14389)  
**Date:** 2023-09-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.14389v1/

## Key points

- Document question answering models have evolved to include a vision encoder and a Large Language Model (LLM)
- The vision encoder captures layout and visual elements in images, while the LLM contextualizes questions with external knowledge
- Effectiveness of an LLM-only approach in document question answering tasks
- Strategies for serializing textual information within document images and feeding it to an instruction-tuned LLM
- Thorough quantitative analysis on the feasibility of this approach across six diverse benchmark datasets using varying scales of LLMs
- Relying solely on the LLM can yield results comparable to state-of-the-art performance on various datasets
- Importance of layout and image content information in document question answering models
- Analyzing example document image question-answer pairs for different types of questions can help understand model potential and overall performance for unseen tasks
- Advancements in document question answering models enhance ability to extract information from complex documents efficiently and accurately

## Summary

In recent years, document question answering models have evolved to include two crucial components: the vision encoder and a Large Language Model (LLM). The vision encoder captures layout and visual elements within images, while the LLM contextualizes questions with external knowledge for accurate answers. However, their relative contributions to the task remain unclear. This study explores three main aspects to delve deeper into this topic: 
1. The effectiveness of an LLM-only approach in document question answering tasks. 2. Strategies for serializing textual information within document images and directly feeding it to an instruction-tuned LLM. 3. A thorough quantitative analysis on the feasibility of such an approach across six diverse benchmark datasets using varying scales of LLMs. The findings suggest that relying solely on the LLM can yield results comparable to state-of-the-art performance on various datasets. This evaluation framework not only provides insights into the importance of layout and image content information but also serves as a valuable resource for selecting appropriate datasets for future research endeavors. Moreover, analyzing example document image question-answer pairs for different types of questions can help researchers understand model potential and overall performance for unseen tasks. This study sheds light on how advancements in document question answering models can enhance our ability to extract information from complex documents efficiently and accurately.