Can Large Language Models Infer and Disagree Like Humans?

Large Language Models (LLMs) have gained significant attention for their impressive performance in various tasks. However, their ability to align with human disagreement distribution and accurately solve Natural Language Inference (NLI) tasks has not been thoroughly studied. In this paper, the authors evaluate the performance and alignment of LLMs with humans using two different techniques: Monte Carlo Reconstruction (MCR) and Log Probability Reconstruction (LPR). The results of the study show that LLMs exhibit limited ability in solving NLI tasks and fail to capture human disagreement distribution effectively. This raises concerns about their natural language understanding (NLU) ability and their representativeness of human users. The authors compare the performances of generative LLMs with other fully fine-tuned smaller models on a fundamental NLI task. Despite variations in model distribution through different reconstruction methods and prompt types, none of the models achieve performances that closely align with the accuracy and disagreement levels observed in the human population. This indicates that further research is needed to improve LLM performance in capturing human distribution at a population level. The study acknowledges some limitations, including the possibility that 100 annotators may not be sufficient to represent the full range of human disagreement distribution accurately. Future studies could benefit from datasets with more diverse label variations to cover a wider range of model types and create evaluation benchmarks for measuring disagreement levels. The authors also discuss a potential reason for LLMs' underperformance in NLI tasks compared to humans. They hypothesize that it may be due to how data was collected, where annotators were asked to predict numerical scores for strength which were later discretized. This discrepancy between continuous scores provided by humans and discretized scores used during training might contribute to the misalignment between LLMs and human performance. In conclusion, this paper highlights the limited ability of billion-scale LLMs in solving NLI tasks and capturing human disagreement distribution effectively. It emphasizes the need for further research to improve LLM performance and understand the latent factors that contribute to disagreement in LLMs compared to humans.

Large Language Models: Evaluating Performance and Alignment with Human Disagreement Distribution

In recent years, large language models (LLMs) have gained significant attention for their impressive performance in various tasks. However, their ability to align with human disagreement distribution and accurately solve natural language inference (NLI) tasks has not been thoroughly studied. In this paper, the authors evaluate the performance and alignment of LLMs with humans using two different techniques: Monte Carlo Reconstruction (MCR) and Log Probability Reconstruction (LPR). The results of the study show that LLMs exhibit limited ability in solving NLI tasks and fail to capture human disagreement distribution effectively. This raises concerns about their natural language understanding (NLU) ability and their representativeness of human users.

Experimental Setup

The authors compare the performances of generative LLMs with other fully fine-tuned smaller models on a fundamental NLI task. They use 100 annotators from Amazon Mechanical Turk who are asked to predict numerical scores for strength which were later discretized into three labels: entailment, neutral or contradiction. To measure model performance against human accuracy levels, they use Matthews Correlation Coefficient (MCC), which is a standard metric used in evaluating binary classifiers such as those used in NLI tasks. To measure model alignment with human disagreement distribution, they use two reconstruction methods: Monte Carlo Reconstruction (MCR) and Log Probability Reconstruction (LPR). MCR reconstructs label distributions by randomly sampling from an underlying probability distribution while LPR reconstructs label distributions by taking log probabilities from each model's output layer before softmax normalization. The authors also consider variations in prompt types including single-sentence prompts vs multi-sentence prompts as well as factual vs counterfactual prompts when evaluating model performance against humans.

Results & Discussion

Despite variations in model distribution through different reconstruction methods and prompt types, none of the models achieve performances that closely align with the accuracy and disagreement levels observed in the human population. This indicates that further research is needed to improve LLM performance in capturing human distribution at a population level. The study acknowledges some limitations, including the possibility that 100 annotators may not be sufficient to represent the full range of human disagreement distribution accurately. Future studies could benefit from datasets with more diverse label variations to cover a wider range of model types and create evaluation benchmarks for measuring disagreement levels.

The authors also discuss a potential reason for LLMs' underperformance in NLI tasks compared to humans - discrepancy between continuous scores provided by humans during annotation process versus discretized scores used during training might contribute to misalignment between LLMs' predictions and actual ground truth labels generated by humans..

Conclusion

In conclusion, this paper highlights the limited ability of billion-scale LLMs in solving NLI tasks and capturing human disagreement distribution effectively. It emphasizes the need for further research to improve LLM performance and understand latent factors that contribute to disagreements between machines’ predictions versus actual ground truth labels generated by humans .