# Debating with More Persuasive LLMs Leads to More Truthful Answers

**Authors:** Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, Ethan Perez  
**arXiv:** [2402.06782](https://arxiv.org/abs/2402.06782)  
**Date:** 2024-02-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.06782v1/

## Key points

- Authors explore alignment of large language models (LLMs) with desired behavior without human-labeled data
- Study evaluates effectiveness of debate method using LLM experts and non-expert to select answers
- Results show engaging in debates improves accuracy for both non-expert models and humans
- Accuracy rates: 76% for non-expert models, 88% for humans, surpassing naive baselines
- Optimizing expert debaters for persuasiveness enhances non-expert's ability to identify truth in debates
- Research supports feasibility of aligning models through debate even without ground truth
- Findings offer innovative approaches to leveraging model interactions for improved performance in AI decision-making

## Summary

In their paper titled "Debating with More Persuasive LLMs Leads to More Truthful Answers," authors Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez explore the alignment of large language models (LLMs) with desired behavior without relying heavily on human-labeled data. The study evaluates the effectiveness of using a debate method where two LLM experts argue for different answers and a non-expert selects the answer. The central question posed is whether weaker models can assess the correctness of stronger models in a scenario where experts possess the necessary information to answer questions while non-experts lack this information. The results show that engaging in debates consistently improves the ability of both non-expert models and humans to answer questions accurately. Specifically, the accuracy rates achieved were 76% for non-expert models and 88% for humans, surpassing naive baselines at 48% and 60%, respectively. Moreover, optimizing expert debaters for persuasiveness through unsupervised methods enhances the non-expert's ability to identify truth in debates. Overall,the findings provide promising empirical evidence supporting the feasibility of aligning models through debate even in situations where ground truth is absent. The authors' research sheds light on innovative approaches to leveraging model interactions for improved performance and accuracy in complex decision-making processes within artificial intelligence systems.