# Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models

**Authors:** Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral  
**arXiv:** [2404.15522](https://arxiv.org/abs/2404.15522)  
**Date:** 2024-04-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.15522v1/

## Key points

- Recent advancements in large language models (LLMs) have shown impressive capabilities in natural language understanding tasks
- Logical reasoning is crucial for question-answering systems and conversational agents
- LogicBench is a dataset specifically evaluating LLMs' logical reasoning using a single inference rule, covering 25 distinct reasoning patterns across propositional logic, first-order logic, and non-monotonic logics
- This work explores non-monotonic reasoning within NLP and includes various inference rules such as hypothetical syllogism and disjunctive syllogism
- Existing LLMs struggle with complex reasoning instances and negations on LogicBench
- The availability of data and code on GitHub facilitates reproducibility and encourages collaboration towards advancing logical reasoning abilities in LLMs

## Summary

Introduction
Recent advancements in large language models (LLMs) have shown impressive capabilities in natural language understanding tasks. However, their ability to engage in logical reasoning remains a topic of ongoing research. Logical reasoning is crucial for question-answering systems and conversational agents. Existing datasets evaluating LLMs' logical reasoning abilities often focus on limited aspects or combine it with other forms of reasoning. To address this gap, we introduce LogicBench - a meticulously designed dataset specifically evaluating LLMs' logical reasoning using a single inference rule. It covers 25 distinct reasoning patterns across propositional logic, first-order logic, and non-monotonic logics. This is the first work exploring non-monotonic reasoning within NLP and includes various inference rules such as hypothetical syllogism and disjunctive syllogism. We also analyze the prevalence of these rules in pre-training data and their impact on model performance. The evaluation tasks include Binary Question-Answering (BQA) and Multiple-Choice Questions-Answering (MCQA). Our results reveal that existing LLMs struggle with complex reasoning instances and negations on LogicBench. By shedding light on these limitations and providing insights into enhancing logical reasoning abilities in LLMs through systematic evaluation, our work paves the way for future research in this area. The availability of data and code on GitHub facilitates reproducibility and encourages collaboration towards advancing logical reasoning abilities in LLMs.