# BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information

**Authors:** Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, Deepak Ramachandran  
**arXiv:** [2306.07934](https://arxiv.org/abs/2306.07934)  
**Date:** 2023-06-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.07934v1/

## Key points

- Automated reasoning with unstructured natural text is important for many NLP and AI applications.
- Existing evaluation assumes consistent information, but real-world information is often inconsistent or contradictory.
- Google Research developed BoardgameQA dataset to measure LM reasoning capacity in the presence of conflicting input sources.
- BoardgameQA incorporates reasoning with implicit background knowledge and scenarios where additional information needs to come from the model itself.
- LMs perform poorly when reasoning with conflicting inputs and additional knowledge is needed.
- Resolving conflicts by adopting the source with higher preference can be effective.
- BoardgameQA highlights an important gap in current LMs' understanding capacity and can guide future work to improve their understanding ability under this setup.

## Summary

Automated reasoning with unstructured natural text is a crucial requirement for many potential applications of Natural Language Processing (NLP) and for developing robust Artificial Intelligence (AI) systems. While Language Models (LMs) have demonstrated complex reasoning capacities even without any finetuning, existing evaluation for automated reasoning assumes access to a consistent and coherent set of information over which models reason. However, in the real-world, available information is frequently inconsistent or contradictory, and therefore models need to be equipped with a strategy to resolve such conflicts when they arise. To address this issue, researchers at Google Research have formulated the problem of reasoning with contradictory information guided by preferences over sources as the classical problem of defeasible reasoning and developed a dataset called BoardgameQA for measuring the reasoning capacity of LMs in this setting. BoardgameQA incorporates reasoning with implicit background knowledge to better reflect reasoning problems in downstream applications. The dataset includes scenarios where knowledge required for reasoning is only partially provided as input and additional information needs to come from the model itself. The researchers benchmarked various LMs on BoardgameQA and observed that LMs perform poorly when reasoning with conflicting inputs. In the case of smaller models, performance was also poor when additional knowledge from the LM was needed. Since reasoning over contradicting and incomplete sets of information is a common scenario in real-world applications, these results highlight an important gap in the current LMs' reasoning capacity. The work spans three dimensions: text-based logical reasoning, reasoning with conflicting sources, and incomplete information. Earlier works on natural language logical reasoning have finetuned LMs to directly provide answers to logical questions. Later work showed that explicitly generating the entire proof leads to substantial improvements both in accuracy and interpretability. One widely-applicable way of resolving conflicts is to impose preferences over information sources based on source credibility or recency and adopt the source with higher preference. In conclusion, BoardgameQA provides a valuable resource for measuring the natural language reasoning ability of LMs in the presence of conflicting input sources. The dataset highlights an important gap in the current LMs' understanding capacity and can guide future work developing methodology to improve their understanding ability under this setup or finding alternative formulations of conflict resolution that better facilitate LM understanding.