# Not All Large Language Models (LLMs) Succumb to the "Reversal Curse": A Comparative Study of Deductive Logical Reasoning in BERT and GPT Models

**Authors:** Jingye Yang, Da Wu, Kai Wang  
**arXiv:** [2312.03633](https://arxiv.org/abs/2312.03633)  
**Date:** 2023-12-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.03633v1/

## Key points

- Study aimed to compare deductive logical reasoning capabilities of BERT and GPT
- Focus on addressing the "Reversal Curse" and its impact on logical deduction
- Utilized existing dataset from LLaMA model, adapted it for training and testing with BERT
- Dataset created by generating new fictitious names using ChatGPT and substituting them into prompts
- 900 prompts with positive and negative labels
- Trained and tested on reverse direction using different name-description pairs for fairness in comparison
- Fine-tuned vanilla pretrained BERT model (bert-base-cased) for text classification tasks
- Measured accuracy in distinguishing between positive and negative prompts
- Explored BERT's abilities in mastering set operations like intersection and union
- Trained encoder and decoder models on two sets, evaluated performance on three newly created sets involving various combinations of union and intersection operations
- Encoder and decoder models excelled in scenarios involving two sets but faced difficulties with operations involving three sets
- Choosing between BERT or GPT should depend on specific task requirements
- Leveraging their respective strengths can lead to more effective results

## Summary

The study aimed to compare the deductive logical reasoning capabilities of two large language models (LLMs), BERT and GPT. Specifically, it focused on addressing the "Reversal Curse" and its impact on logical deduction. To conduct their analysis, the researchers utilized an existing dataset from the LLaMA model and prepared it for training and testing with BERT. The dataset was adapted by creating new fictitious names using ChatGPT and randomly substituting them into positive prompts. This resulted in 900 prompts with both positive and negative labels. To ensure fairness in comparison, the researchers also trained and tested on the reverse direction using different name-description pairs. They fine-tuned a vanilla pretrained BERT model (bert-base-cased) specifically for text classification tasks and measured accuracy based on its ability to distinguish between positive and negative prompts. In addition to comparing BERT's performance with GPT in deductive reasoning tasks, the researchers also explored its abilities in mastering set operations such as intersection and union. They trained encoder and decoder models on two sets and evaluated their performance on three newly created sets involving various combinations of union and intersection operations. The findings revealed that while both encoder and decoder models excelled in scenarios involving two sets, they encountered difficulties when dealing with operations involving three sets. This highlights the distinct characteristics of these models in simple versus complex logical reasoning tasks. In conclusion, this study emphasizes that choosing between BERT or GPT should depend on the specific requirements of the task at hand. Leveraging their respective strengths in bidirectional context comprehension and sequence prediction can lead to more effective results.