# Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

**Authors:** Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica  
**arXiv:** [2306.05685](https://arxiv.org/abs/2306.05685)  
**Date:** 2023-06-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.05685v4/

## Key points

- Authors address the challenge of evaluating large language model (LLM) based chat assistants due to diverse capabilities and limitations of existing benchmarks
- Researchers explore the use of robust LLMs as judges to assess models on open-ended questions
- Study discusses issues such as position bias and verbosity bias in LLM-as-a-judge method
- Authors propose solutions to mitigate biases for a more reliable evaluation process
- Introduction of two benchmarks: MT-bench (multi-turn question set) and Chatbot Arena (crowdsourced battle platform) to verify alignment between LLM judges and human preferences
- Results show that powerful LLM judges like GPT-4 can effectively match controlled and crowdsourced human preferences with an agreement rate exceeding 80%
- Employing LLM-as-a-judge approach is scalable and interpretable way to approximate costly-to-obtain human preferences
- Evaluation of several variants of LLaMA and Vicuna using new benchmark alongside traditional benchmarks shows complementarity of different evaluation methods
- MT-bench questions, 3K expert votes, and 30K conversations reflecting human preferences are made publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge

## Summary

In the study titled "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," authors Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica address the challenge of evaluating large language model (LLM) based chat assistants due to their diverse capabilities and the limitations of existing benchmarks in capturing human preferences accurately. To overcome this challenge, the researchers explore the use of robust LLMs as judges to assess these models on more open-ended questions. The study delves into the utilization and constraints of LLM-as-a-judge method by discussing issues such as position bias and verbosity bias among others. The authors propose solutions to mitigate some of these biases for a more reliable evaluation process. By introducing two benchmarks - MT-bench which consists of a multi-turn question set and Chatbot Arena which is a crowdsourced battle platform - they aim to verify the alignment between LLM judges and human preferences. The results demonstrate that powerful LLM judges like GPT-4 can effectively match both controlled and crowdsourced human preferences with an agreement rate exceeding 80%, similar to the level of agreement observed among humans. This suggests that employing LLM-as-a-judge approach is a scalable and interpretable way to approximate human preferences which are otherwise costly to obtain. Furthermore, by evaluating several variants of LLaMA and Vicuna using their benchmark alongside traditional benchmarks,the researchers show how these different evaluation methods complement each other. The study makes its MT-bench questions along with 3K expert votes and 30K conversations reflecting human preferences publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge. Overall, this research provides valuable insights into enhancing the evaluation process for large language model-based chat assistants through innovative approaches like using strong LLMs as judges along with comprehensive benchmarks like MT-Bench and Chatbot Arena.