# DARE: Diverse Visual Question Answering with Robustness Evaluation

**Authors:** Hannah Sterz, Jonas Pfeiffer, Ivan Vulić  
**arXiv:** [2409.18023](https://arxiv.org/abs/2409.18023)  
**Date:** 2024-09-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.18023v1/

## Key points

- Authors explore capabilities and limitations of Vision Language Models (VLMs)
- VLMs excel in standard image classification and image-text matching tasks
- Challenges in crucial vision-language reasoning abilities like counting and spatial reasoning
- Existing benchmarks do not adequately assess the robustness of VLMs
- Introduction of DARE, a multiple-choice Visual Question Answering (VQA) benchmark to evaluate VLM performance across five diverse categories
- State-of-the-art VLMs struggle with questions across most categories and fail to consistently deliver peak performance under different robustness evaluations
- Open-source VLMs exhibit lower robustness compared to closed-source models
- Need for improved evaluation metrics to accurately assess the robustness of VLMs

## Summary

, , , , 

In their paper titled "DARE: Diverse Visual Question Answering with Robustness Evaluation," authors Hannah Sterz, Jonas Pfeiffer, and Ivan Vulić explore the capabilities and limitations of Vision Language Models (VLMs). These models combine text-only large language models with vision-only models to process multi-modal vision-text input. While VLMs excel in standard image classification and image-text matching tasks, they face challenges in crucial vision-language reasoning abilities like counting and spatial reasoning. The authors highlight that existing benchmarks do not adequately assess the robustness of VLMs, leading to inconsistencies in performance under varying conditions. To address this gap, the authors introduce DARE, a comprehensive multiple-choice Visual Question Answering (VQA) benchmark designed to evaluate VLM performance across five diverse categories. DARE includes four robustness-oriented evaluations that test variations in prompts, answer options subsets, output format, and correct answers. Through their experiments, the authors find that state-of-the-art VLMs struggle with questions across most categories and fail to consistently deliver peak performance under different robustness evaluations. In fact, the worst-case performance can be up to 34% lower than in standard scenarios. Furthermore, the study reveals that open-source VLMs like LLaVA 1.6 and Idefics2 exhibit lower robustness compared to closed-source models such as GPT-4 and Gemini. However, even these closed-source models demonstrate brittleness when faced with diverse variations in inputs. Overall, the findings underscore the need for improved evaluation metrics to assess the robustness of VLMs accurately and enhance their performance across challenging vision-language reasoning tasks.