# Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models

**Authors:** Chaoya Jiang, Wei Ye, Mengfan Dong, Hongrui Jia, Haiyang Xu, Ming Yan, Ji Zhang, Shikun Zhang  
**arXiv:** [2402.15721](https://arxiv.org/abs/2402.15721)  
**Date:** 2024-02-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.15721v1/

## Key points

- Large Vision Language Models (LVLMs) face challenges with hallucinations, including inconsistencies between images and descriptions
- Previous research has focused on hallucinations related to objects, attributes, and relations in LVLMs but overlooked complex narrative-based hallucinations
- Authors introduce a refined taxonomy of hallucinations that includes a new category: Event Hallucination
- Utilizing advanced LVLMs, authors generate and filter fine-grained hallucinatory data with a focus on event hallucinations
- Proposed benchmark aims to assess LVLMs' ability to handle various types of hallucinations effectively
- Authors provide a reliable tool for evaluating LVLMs' efficacy in addressing hallucination issues through their taxonomy and evaluation framework
- Plan to release code and data for further research and development in this area
- Emphasize the importance of annotating different types of hallucinations for enhanced understanding and evaluation processes

## Summary

The paper "Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models" by Chaoya Jiang et al. addresses the challenges faced by Large Vision Language Models (LVLMs) in dealing with hallucinations. These inconsistencies between images and their descriptions have been previously identified in relation to objects, attributes, and relations in LVLMs. However, complex hallucinations that construct an entire narrative around a fictional entity have often been overlooked. To tackle this issue, the authors introduce a refined taxonomy of hallucinations that includes a new category: Event Hallucination. By utilizing advanced LVLMs, the authors generate and filter fine-grained hallucinatory data encompassing various types of hallucinations with a specific focus on event hallucinations. This approach lays the foundation for integrating discriminative and generative evaluation methods within a universal evaluation framework. The proposed benchmark aims to assess LVLMs' ability to handle a broad spectrum of hallucinations effectively. Through their innovative taxonomy and evaluation framework, the authors provide a reliable tool for evaluating LVLMs' efficacy in addressing hallucination issues. They also plan to release their code and data for further research and development in this area. In addition to discussing generative evaluation methods currently used in assessing models' performance based on generating hallucinatory content, the paper emphasizes the importance of annotating different types of hallucinations to enhance understanding and evaluation processes. Overall,"Hal-Eval" presents a significant contribution to improving the assessment of LVLMs' capabilities in handling complex hallucinations through its comprehensive taxonomy and evaluation framework.