# MultEval: Supporting Collaborative Alignment for LLM-as-a-Judge Evaluation Criteria

**Authors:** Charles Chiang, Simret Gebreegziabher, Annalisa Szymanski, Yukun Yang, Hyo Jin Do, Zahra Ashktorab, Werner Geyer, Toby Li, Diego Gomez-Zara  
**arXiv:** [2604.26679](https://arxiv.org/abs/2604.26679)  
**Date:** 2026-04-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2604.26679v1/

## Key points

- LLM-as-a-judge approaches are gaining traction as a scalable solution for evaluating model behaviors
- A key challenge is the creation of evaluation criteria, often crafted by a single individual and reflecting biases
- Defining criteria requires a complex and collaborative process involving multiple stakeholders with diverse values, interpretations, and priorities
- Existing tools do not adequately support collaborative development of evaluation criteria
- Formative study explored how stakeholders can collaboratively develop, negotiate, and refine evaluation criteria for LLM-as-a-judge systems
- Challenges include establishing shared understanding among stakeholders with varying expertise and priorities, aligning values across different parties, and translating human judgments into interpretable criteria for LLM judges
- MultEval system introduced to facilitate collaborative criteria development through consensus-building theory and iterative revision while maintaining transparency in encoding judgments
- Case study demonstrated how domain experts utilized MultEval to collaboratively author evaluation criteria, showcasing the importance of coordination and consensus-building in creating robust evaluation frameworks for LLM-as-a-judge systems

## Summary

In the realm of evaluating model behaviors, LLM-as-a-judge approaches have gained traction as a scalable solution. However, a key challenge lies in the creation of evaluation criteria. These criteria are often crafted by a single individual and may reflect their biases and perspectives. In reality, defining such criteria is a complex and collaborative process that involves multiple stakeholders with diverse values, interpretations, and priorities. Unfortunately, existing tools do not adequately support this aspect. To delve deeper into this issue, a formative study was conducted to explore how stakeholders can collaboratively develop, negotiate, and refine evaluation criteria for LLM-as-a-judge systems. The findings from this study shed light on the hurdles faced in human oversight. These challenges include establishing shared understanding among stakeholders with varying expertise and priorities, aligning values across different parties, and translating nuanced human judgments into criteria that are both interpretable and actionable for LLM judges. Building upon these insights, the researchers introduced MultEval—a system designed to facilitate collaborative criteria development. MultEval enables multiple evaluators to identify and address disagreements using consensus-building theory. It also allows for iterative revision of criteria while maintaining a history of proposals and examples attached to them. Additionally, it ensures transparency in encoding judgments into an automated evaluator. Furthermore, a case study was presented where a team of domain experts utilized MultEval to collaboratively author evaluation criteria. This case study exemplified how coordination and consensus-building among stakeholders shape the evolution of evaluation criteria. The research culminated in showcasing the importance of collaborative alignment in creating robust evaluation frameworks for LLM-as-a-judge systems.