# Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

**Authors:** Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, Patrick Lewis  
**arXiv:** [2404.18796](https://arxiv.org/abs/2404.18796)  
**Date:** 2024-04-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.18796v1/

## Key points

- Authors address the challenge of evaluating Large Language Models (LLMs) due to their rapid advancement
- Difficulty in finding suitable data to assess specific model properties and evaluating the accuracy of a model's freeform generation in isolation
- Proposal to use Panel of Diverse Models (PoLL) composed of multiple smaller models instead of relying on a single large judge model like GPT4
- Experiments across three distinct judge settings and six different datasets show that PoLL outperforms a single large judge model
- PoLL approach exhibits less intra-model bias and is more cost-effective
- Embracing collaborative evaluation through PoLLs can enhance accuracy and efficiency in assessing LLM performance

## Summary

In their paper titled "Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models," authors Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis address the challenge of accurately evaluating the quality of Large Language Models (LLMs) in the face of their rapid advancement. The authors highlight the difficulty in finding suitable data to assess specific model properties and emphasize the complexity of evaluating the accuracy of a model's freeform generation in isolation. To tackle this issue, many evaluations now utilize LLMs themselves as judges to evaluate the quality of outputs from other LLMs. are difficult to assess due to  in Large Language Models (LLMs). This poses a challenge for researchers trying to  LLM performance accurately. In response to this issue, authors Pat Verga et al. propose using a , composed of multiple smaller models instead of relying on a single large judge model like GPT4. The authors conducted experiments across three distinct judge settings and six different datasets to demonstrate that utilizing a PoLL outperforms a single large judge model. The PoLL approach exhibits less intra-model bias due to its composition of disjoint model families and is also significantly more cost-effective. By moving away from reliance on individual large models and embracing collaborative evaluation through PoLLs, researchers can enhance the accuracy and efficiency of assessing LLM performance in various contexts. Overall, offers an innovative solution to evaluating LLM generations and addresses the drawbacks of using a single large judge model, such as high costs and intramodel bias.