# Can Large Language Models Infer and Disagree Like Humans?

**Authors:** Noah Lee, Na Min An, James Thorne  
**arXiv:** [2305.13788](https://arxiv.org/abs/2305.13788)  
**Date:** 2023-05-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.13788v1/

## Key points

- Large Language Models (LLMs) have impressive performance in various tasks
- LLMs' ability to align with human disagreement distribution and solve Natural Language Inference (NLI) tasks is not well-studied
- Authors evaluate LLMs' performance and alignment using Monte Carlo Reconstruction (MCR) and Log Probability Reconstruction (LPR)
- LLMs exhibit limited ability in solving NLI tasks and fail to capture human disagreement distribution effectively
- Concerns raised about their natural language understanding (NLU) ability and representativeness of human users
- Generative LLMs compared with fully fine-tuned smaller models on an NLI task, but none achieve performances closely aligned with human accuracy and disagreement levels
- Further research needed to improve LLM performance in capturing human distribution at a population level
- Limitations include the possibility that 100 annotators may not be sufficient to represent full range of human disagreement distribution accurately
- Future studies could benefit from datasets with more diverse label variations to create evaluation benchmarks for measuring disagreement levels
- Discrepancy between continuous scores provided by humans and discretized scores used during training may contribute to misalignment between LLMs and human performance in NLI tasks
- Limited ability of billion-scale LLMs highlighted in solving NLI tasks and capturing human disagreement distribution effectively
- Need for further research to improve LLM performance and understand factors contributing to disagreement in LLMs compared to humans.

## Summary

Large Language Models (LLMs) have gained significant attention for their impressive performance in various tasks. However, their ability to align with human disagreement distribution and accurately solve Natural Language Inference (NLI) tasks has not been thoroughly studied. In this paper, the authors evaluate the performance and alignment of LLMs with humans using two different techniques: Monte Carlo Reconstruction (MCR) and Log Probability Reconstruction (LPR). The results of the study show that LLMs exhibit limited ability in solving NLI tasks and fail to capture human disagreement distribution effectively. This raises concerns about their natural language understanding (NLU) ability and their representativeness of human users. The authors compare the performances of generative LLMs with other fully fine-tuned smaller models on a fundamental NLI task. Despite variations in model distribution through different reconstruction methods and prompt types, none of the models achieve performances that closely align with the accuracy and disagreement levels observed in the human population. This indicates that further research is needed to improve LLM performance in capturing human distribution at a population level. The study acknowledges some limitations, including the possibility that 100 annotators may not be sufficient to represent the full range of human disagreement distribution accurately. Future studies could benefit from datasets with more diverse label variations to cover a wider range of model types and create evaluation benchmarks for measuring disagreement levels. The authors also discuss a potential reason for LLMs' underperformance in NLI tasks compared to humans. They hypothesize that it may be due to how data was collected, where annotators were asked to predict numerical scores for strength which were later discretized. This discrepancy between continuous scores provided by humans and discretized scores used during training might contribute to the misalignment between LLMs and human performance. In conclusion, this paper highlights the limited ability of billion-scale LLMs in solving NLI tasks and capturing human disagreement distribution effectively. It emphasizes the need for further research to improve LLM performance and understand the latent factors that contribute to disagreement in LLMs compared to humans.