# Confidence Improves Self-Consistency in LLMs

**Authors:** Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, Gal Yona  
**arXiv:** [2502.06233](https://arxiv.org/abs/2502.06233)  
**Date:** 2025-02-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.06233v1/

## Key points

- Introduction of Confidence-Informed Self-Consistency (CISC) as a decoding strategy to enhance Large Language Models (LLMs) performance on reasoning tasks
- CISC uses confidence scores obtained directly from the model to implement a weighted majority vote, prioritizing high-confidence paths for identifying correct answers with smaller sample sizes
- Experiments showed that CISC outperformed traditional self-consistency in various configurations, reducing required reasoning paths by over 40% on average
- Within-question confidence evaluation introduced for improved accuracy, showing standard evaluation methods are poor predictors of success in distinguishing correct and incorrect answers
- Qualitative analysis revealed agreement between model confidence scores and human assessments of reasoning paths' quality, suggesting LLMs can self-assess their responses

## Summary

, , , , 

In this study, the researchers introduce Confidence-Informed Self-Consistency (CISC) as a decoding strategy to enhance the performance of Large Language Models (LLMs) on reasoning tasks. The traditional self-consistency decoding method is effective but computationally expensive, requiring sampling of numerous reasoning paths to increase the chances of selecting the correct answer. CISC addresses this issue by implementing a weighted majority vote based on confidence scores obtained directly from the model. By prioritizing high-confidence paths, CISC can identify the correct answer with a significantly smaller sample size, reducing computational costs. The researchers conducted experiments comparing CISC and self-consistency across various confidence extraction methods, reasoning tasks, and LLM models. They found that CISC outperformed self-consistency in nearly all configurations, reducing the required number of reasoning paths by over 40% on average. Additionally, they introduced within-question confidence evaluation for improved accuracy, demonstrating that standard evaluation methods are poor predictors of success in distinguishing correct and incorrect answers to the same question. Furthermore, a qualitative analysis revealed a significant agreement between model confidence scores and human assessments of reasoning paths' quality. Responses identified by the model as low-confidence were more likely to be flagged by human evaluators as exhibiting signs of low-quality reasoning patterns. This suggests that LLMs are capable of self-assessing their responses. Overall, this study contributes practical methods and foundational insights in the field of natural language processing. It not only proposes CISC as an efficient alternative to self-consistency for LLMs but also introduces within-question confidence evaluation for improved accuracy and provides empirical evidence supporting LLMs' ability to self-assess their outputs. These findings have practical implications for improving LLM performance and contribute to ongoing debates about LLM capabilities in judging the correctness of their own outputs.