# Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

**Authors:** Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, Yue Zhang  
**arXiv:** [2304.03439](https://arxiv.org/abs/2304.03439)  
**Date:** 2023-04-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03439v3/

## Key points

- The paper is titled "Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4"
- The authors investigate natural language understanding and logical reasoning
- They analyze multiple datasets with popular benchmarks like LogiQA and ReClor, as well as newly-released datasets like AR-LSAT
- The authors test multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning
- Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks
- With early access to the GPT-4 API, they were able to conduct intense experiments on the model which showed that it yields even higher performance on most logical reasoning datasets
- ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor; however, their performance drops significantly when handling newly released and out-of-distribution datasets
- Logical reasoning remains challenging for both models, especially on out-of-distribution and natural language inference datasets
- To aid future research in this area, the authors release prompt-style logical reasoning datasets as a benchmark suite named LogiEval

## Summary

In their paper titled "Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4," authors Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang delve into the topic of natural language understanding and logical reasoning. The release of Generative Pretrained Transformer 4 (GPT-4) has been touted as a significant advancement in reasoning tasks, prompting the authors to investigate its performance on various logical reasoning datasets. The report analyzes multiple datasets with popular benchmarks like LogiQA and ReClor, as well as newly-released datasets like AR-LSAT. The authors test multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. They also construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4 while making a performance comparison between them. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API, they were able to conduct intense experiments on the model which showed that it yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor; however, their performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for both models, especially on out-of-distribution and natural language inference datasets. To aid future research in this area, the authors release prompt-style logical reasoning datasets as a benchmark suite named LogiEval. Overall, this paper provides valuable insights into the current state of natural language understanding with respect to logical reasoning ability using advanced models like ChatGPT and GPT-4 while highlighting areas for further improvement in this field.