# Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

**Authors:** Sangwoo Cho, Kushal Chawla, Pengshan Cai, Zefang Liu, Chenyang Zhu, Shi-Xiong Zhang, Sambit Sahu  
**arXiv:** [2606.27226](https://arxiv.org/abs/2606.27226)  
**Date:** 2026-06-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2606.27226v1/

## Key points

- Authors address challenges in evaluating outputs from Large Language Models (LLMs) in NLP
- Introduce BINEVAL as a novel framework to overcome limitations in traditional human evaluation methods and existing lexical metrics
- BINEVAL breaks down evaluation criteria into atomic binary questions for multi-dimensional scores
- Utilizes task prompts and meta-prompts for fine-grained evaluation questions
- Provides transparent feedback at the question level and produces calibrated overall scores
- Decomposition of evaluation criteria makes it easier to inspect, diagnose evaluations, and facilitate prompt improvement
- Demonstrated effectiveness across various tasks like SummEval, Topical-Chat, and QAGS
- Compared against strong baselines like UniEval and G-Eval with superior performance on factual consistency benchmarks like QAGS
- Exhibits competitive correlation with human judgments while better aligning with human score distributions
- Avoids common ceiling effects observed in previous LLM evaluators, leading to improved discrimination between borderline cases and clearly flawed outputs
- Supports iterative prompt optimization for summarization tasks and generation prompts under self-update and cross-model update scenarios

## Summary

In their paper titled "Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement," authors Sangwoo Cho, Kushal Chawla, Pengshan Cai, Zefang Liu, Chenyang Zhu, Shi-Xiong Zhang, and Sambit Sahu address the challenges associated with evaluating outputs from Large Language Models (LLMs) in Natural Language Processing (NLP). They introduce BINEVAL as a novel framework to overcome limitations in traditional human evaluation methods and existing lexical metrics. <br>
BINEVAL breaks down evaluation criteria into atomic binary questions and aggregates them to generate multi-dimensional scores that are easily interpretable. By utilizing a task prompt to generate fine-grained evaluation questions through a meta-prompt, BINEVAL enables LLMs to independently answer each question for every output. This approach provides transparent feedback at the question level and produces calibrated overall scores. <br>
The decomposition of evaluation criteria in BINEVAL makes it easier to inspect and diagnose evaluations while also facilitating prompt improvement by offering actionable insights based on the feedback received. The authors demonstrate the effectiveness of BINEVAL across various tasks such as SummEval, Topical-Chat, and QAGS. They compare BINEVAL against strong baselines like UniEval and G-Eval and show superior performance on factual consistency benchmarks like QAGS. <br>
Furthermore, BINEVAL exhibits competitive correlation with human judgments while better aligning with human score distributions. It avoids common ceiling effects observed in previous LLM evaluators, leading to improved discrimination between borderline cases and clearly flawed outputs. The authors also illustrate how question-level feedback from BINEVAL supports iterative prompt optimization by showcasing enhancements in evaluator prompts for summarization tasks and generation prompts for IFBench under both self-update and cross-model update scenarios. <br>
Overall, BINEVAL stands out as a task-agnostic, training-free, and interpretable evaluation framework that combines robust empirical performance with practical diagnostic capabilities and optimization potential. The authors' work sheds light on innovative approaches to address the challenges of evaluating LLM outputs in NLP effectively.