# Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

**Authors:** Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, Yue Zhang  
**arXiv:** [2304.03439](https://arxiv.org/abs/2304.03439)  
**Date:** 2023-04-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03439v1/

## Key points

- Logical reasoning is essential for natural language understanding
- GPT-4 is a new model that has been highlighted as "advanced" in reasoning tasks
- Multiple logical reasoning datasets were analyzed, including LogiQA, ReClor, and AR-LSAT
- The study tested multi-choice reading comprehension and natural language inference tasks with benchmarks that require logical reasoning
- A logical reasoning out-of-distribution dataset was constructed to investigate the robustness of ChatGPT and GPT-4
- ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks
- GPT-4 shows even higher performance on manual tests conducted by the researchers
- Both models perform relatively well on well-known datasets like LogiQA and ReClor but struggle with newly released and out-of-distribution datasets.

## Summary

The ability to harness logical reasoning is an essential aspect of natural language understanding. With the recent release of Generative Pretrained Transformer 4 (GPT-4), which has been highlighted as "advanced" in reasoning tasks, there is a growing interest in evaluating its performance on various logical reasoning tasks. In this report, multiple logical reasoning datasets are analyzed, including popular benchmarks like LogiQA and ReClor, as well as newly-released datasets like AR-LSAT. The study tests multi-choice reading comprehension and natural language inference tasks with benchmarks that require logical reasoning. To investigate the robustness of ChatGPT and GPT-4, a logical reasoning out-of-distribution dataset was constructed. A performance comparison between ChatGPT and GPT-4 was also made. The experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. Moreover, GPT-4 shows even higher performance on manual tests conducted by the researchers. While ChatGPT and GPT-4 perform relatively well on well-known datasets like LogiQA and ReClor, their performance drops significantly when handling newly released and out-of-distribution datasets. This suggests that logical reasoning remains challenging for both models, especially when dealing with out-of-distribution and natural language inference datasets.