Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   Who can we trust? LLM-as-a-jury for Comparative Assessment

   Meng-Jie Qian, Guangzhi Sun, M. Gales et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 9 times · Not summarized yet

   Abstract

   Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and evaluation aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address t…

2. Very close Cites this paper Not relevant

   Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments

   Yuran Li, J. Mohamud, Chong-Ren Sun et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 22 times · Not summarized yet

   Abstract

   Large language models (LLMs) are being widely applied across various fields, but as tasks become more complex, evaluating their responses is increasingly challenging. Compared to human evaluators, the use of LLMs to support performance evaluation offers a more efficient alternative. However, most studies focus mainly on aligning LLMs' judgments with human preferences, overlooking the existence of biases and mistakes in human judgment. Furthermore, how to select suitable LLM judgments given multiple potential LLM responses remains underexplored. To address these two aforementioned issues, we propose a three-stage meta-judge selection pipeline: 1) developing a comprehensive rubric with GPT-4 a…

3. Very close Cites this paper Uses this paper in depth Not relevant

   SLMJury: Can Small Language Models Judge as Well as Large Ones?

   Anish Laddha, Nitesh Pradhan, Gaurav Srivastava · 2026 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability. We introduce SLMJury, a framework for evaluating small language models (SLMs) as judges across two paradigms: closed-ended binary correctness and open-ended quality scoring. We benchmark 16 SLM judges (0.6B-14B parameters) from four model families across ten benchmarks: eight closed-ended tasks spanning mathematical, scientific, and general reasoning (N=64,824 judgments per configuration), plus SummEval and MT-Bench for summarization and conversational scoring. We formalize judging as a budget-conditioned function and study five dimensions. Four fi…

4. Very close Cites this paper Not relevant

   How Reliable is Multilingual LLM-as-a-Judge?

   Xiyan Fu, Wei Liu · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 47 times · Not summarized yet

   Abstract

   LLM-as-a-Judge has emerged as a popular evaluation strategy, where advanced large language models assess generation results in alignment with human instructions. While these models serve as a promising alternative to human annotators, their reliability in multilingual evaluation remains uncertain. To bridge this gap, we conduct a comprehensive analysis of multilingual LLM-as-a-Judge. Specifically, we evaluate five models from different model families across five diverse tasks involving 25 languages. Our findings reveal that LLMs struggle to achieve consistent judgment results across languages, with an average Fleiss' Kappa of approximately 0.3, and some models performing even worse. To inves…

5. Very close Not relevant

   LLM-as-a-Judge Failures at Automating the Identification of Poor Quality Outputs in Free-Form Texts

   Zong-Xia Li, Xi-Yang Wu, I. Mondal et al. · 2025 · Proceedings of The 5th New Frontiers in Summarization Workshop · cited 3 times · Not summarized yet

   Abstract

   Large language models ( LLM ) such as GPT - 4, Claude, and LLaMA are routinely used to evaluate long-form text generated by language models. We study the ability of these models to identify low-quality texts, an increasingly rare subset of output that is of great interest to pinpoint during development. We present experiments with a panel of LLM judges, and crowd-sourced approximations of reference judgments. Pinpointing sub-par outputs is a difficult task for both models and crowdwork-ers, with models doing better overall. More-over, unlike findings in prior work on factoid QA , panels of cheaper models do not agree as well with high-quality developer judgments of low quality as panels of f…

6. Very close Not relevant

   LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

   Haitao Li, Qian Dong, Junjie Chen et al. · 2024 · Computation and Language · preprint, not peer reviewed · cited 667 times · AI summary available

   Abstract

   The rapid advancement of Large Language Models (LLMs) has driven their expanding application across various fields. One of the most promising applications is their role as evaluators based on natural language responses, referred to as ''LLMs-as-judges''. This framework has attracted growing attention from both academia and industry due to their excellent effectiveness, ability to generalize across tasks, and interpretability in the form of natural language. This paper presents a comprehensive survey of the LLMs-as-judges paradigm from five key perspectives: Functionality, Methodology, Applications, Meta-evaluation, and Limitations. We begin by providing a systematic definition of LLMs-as-Jud…

7. Very close Cited by this paper A method it uses Not relevant

   JudgeLM: Fine-tuned Large Language Models are Scalable Judges

   Liang-Hui Zhu, Xing-Gang Wang, Xin-Long Wang · 2023 · International Conference on Learning Representations · cited 456 times · Not summarized yet

   Abstract

   Evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively. To address this problem, we propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks. We first propose a comprehensive, large-scale, high-quality dataset containing task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges, as well as a new benchmark for evaluating the judges. We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors. We then an…

8. Very close Not relevant

   Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

   Junjie Chen, Yu-Xin Dong, Hai-Tao Li et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 4 times · Not summarized yet

   Abstract

   As large language models (LLMs) are increasingly used for long-form generation, reliably evaluating long-form outputs has become a critical challenge. LLM-as-a-judge offers a scalable alternative to human evaluation, yet its reliability in long-form output evaluation remains underexamined: existing meta-evaluation benchmarks focus mainly on short-form outputs. Compared with short-form evaluation, long-form evaluation is not merely a matter of output length; it often requires judges to make more complex document-level assessments of overall organization, task-relevant coverage and depth, cross-section consistency, and scenario-specific quality criteria. In this work, we introduce LongJudgeBen…

Prior work it builds on

1. Very close Used in depth by this paper Not relevant

   Judging LLM-as-a-judge with MT-Bench and Chatbot Arena

   Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. · 2023 · Computation and Language · cited 12,776 times · Not summarized yet

   Abstract

   Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, such as position and verbosity biases and limited reasoning ability, and propose solutions to migrate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like …

2. Very close A method it uses Not relevant

   PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

   Ruosen Li, Teerth Patel, Xinya Du · 2023 · Trans. Mach. Learn. Res. · cited 149 times · Not summarized yet

   Abstract

   Nowadays, the quality of responses generated by different modern large language models (LLMs) is hard to evaluate and compare automatically. Recent studies suggest and predominantly use LLMs for reference-free evaluation of open-ended question answering. More specifically, they use the recognized"strongest"LLM as the evaluator, which conducts pairwise comparisons of candidate models' answers and provides a ranking score. However, this intuitive method has multiple problems, such as bringing in self-enhancement (favoring its own answers) and positional bias. We draw insights and lessons from the educational domain (Cho&MacArthur, 2011; Walsh, 2014) to improve LLM-based evaluations. Specifical…

3. Very close Not relevant

   LLM Evaluators Recognize and Favor Their Own Generations

   Arjun Panickssery, Samuel R. Bowman, Shi Feng · 2024 · Computation and Language · cited 984 times · AI summary available

   Abstract

   Self-evaluation using large language models (LLMs) has proven valuable not only in benchmarking but also methods like reward modeling, constitutional AI, and self-refinement. But new biases are introduced due to the same LLM acting as both the evaluator and the evaluatee. One such bias is self-preference, where an LLM evaluator scores its own outputs higher than others' while human annotators consider them of equal quality. But do LLMs actually recognize their own outputs when they give those texts higher scores, or is it just a coincidence? In this paper, we investigate if self-recognition capability contributes to self-preference. We discover that, out of the box, LLMs such as GPT-4 and Ll…

4. Close* A method it uses Not relevant

   Leveraging Large Language Models for NLG Evaluation: A Survey

   Zhen Li, Xiaohan Xu, Tao Shen et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 58 times · Not summarized yet
5. Related Used in depth by this paper A method it uses Not relevant

   Measuring and Narrowing the Compositionality Gap in Language Models

   Ofir Press, Muru Zhang, Sewon Min et al. · 2022 · Conference on Empirical Methods in Natural Language Processing · cited 1,589 times · AI summary available

   Abstract

   We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems. We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap. We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining. In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decr…

Papers that cite it

1. Very close Uses this paper in depth Not relevant

   LLMs Judging LLMs: A Simplex Perspective

   Patrick Vossler, Fan Xia, Yifan Mai et al. · 2025 · cited 5 times · Not summarized yet

   Abstract

   Given the challenge of automatically evaluating free-form outputs from large language models (LLMs), an increasingly common solution is to use LLMs themselves as the judging mechanism, without any gold-standard scores. Implicitly, this practice accounts for only sampling variability (aleatoric uncertainty) and ignores uncertainty about judge quality (epistemic uncertainty). While this is justified if judges are perfectly accurate, it is unclear when such an approach is theoretically valid and practically robust. We study these questions for the task of ranking LLM candidates from a novel geometric perspective: for $M$-level scoring systems, both LLM judges and candidates can be represented a…

2. Very close Uses this paper in depth Not relevant

   RoPoLL: Robust Panel of LLM Judges

   Anish Acharya, Kris W Pan, Brian Verkhovsky · 2026 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   The LLM Jury, a Panel of LLM Evaluators (PoLL) reporting consensus scores, has become a practical alternative to single-judge LLM evaluation, yet its statistical behavior remains poorly understood. We formalize the LLM Jury under the Huber contamination model and show that PoLL incurs unbounded bias under any positive contamination, regardless of jury size, whenever a single judge fails in a biased, LLM-typical way (mode collapse, sycophancy, safety refusal). Framing jury consensus as classical robust mean estimation, we propose RoPoLL (Robust Panel of LLM-as-Judge), which preserves the PoLL panel but replaces the aggregation function with a robust mean estimator, instantiated with the geome…

3. Very close Uses this paper in depth Not relevant

   Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process

   Zhijun Chen, Zeyu Ji, Qian-Ren Mao et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet

   Abstract

   We propose LLM-PeerReview, an unsupervised LLM Ensemble method that selects the most ideal response from multiple LLM-generated candidates for each query, harnessing the collective wisdom of multiple models with diverse strengths. LLM-PeerReview is built on a novel, peer-review-inspired framework that offers a transparent and interpretable mechanism, while remaining fully unsupervised for flexible adaptability and generalization. Specifically, it operates in three stages: For scoring, we use the emerging LLM-as-a-Judge technique to evaluate each response by reusing multiple LLMs at hand; For reasoning, we can apply a straightforward averaging strategy or a principled graphical model-based tr…

4. Very close Uses this paper in depth Not relevant

   Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

   José Pombal, Ricardo Rei, Andr'e F. T. Martins · 2026 · arXiv.org · preprint, not peer reviewed · cited 19 times · Not summarized yet

   Abstract

   LLM-as-a-judge has become the de facto approach for evaluating LLM outputs. However, judges are known to exhibit self-preference bias (SPB): they tend to favor outputs produced by themselves or by models from their own family. This skews evaluations and, thus, hinders model development, especially in settings of recursive self-improvement. We present the first study of SPB in rubric-based evaluation, an increasingly popular benchmarking paradigm where judges issue binary verdicts on individual evaluation criteria, instead of assigning holistic scores or rankings. Using IFEval and LiveCodeBench, benchmarks with programmatically verifiable rubrics, we show that SPB persists even when evaluatio…

5. Very close Not relevant

   Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analyses

   Xanh Ho, Jiahao Huang, Florian Boudin et al. · 2025 · IEEE Games Entertainment Media Conference · cited 19 times · Not summarized yet

   Abstract

   Extractive QA tasks are commonly evaluated using Exact Match (EM) and F1-score, but these metrics often fail to reflect true model performance. Recent studies have proposed using large language models (LLMs) as judges (LLM-as-a-judge), yet they often lack comprehensive evaluation across datasets and overlook key factors such as sensitivity to answer types, prompt variations, and self-preference bias. In this work, we conduct a systematic study of LLM-as-a-judge across four extractive QA datasets and various prompt variations, assessing multiple LLM families in both answering and judging roles. Our results show that LLM-as-a-judge judgments correlate much more strongly with human evaluations …