# FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

**Authors:** Bo Yang, Lanfei Feng, Yunkui Chen, Yu Zhang, Xiao Xu, Shijian Li  
**arXiv:** [2602.06625](https://arxiv.org/abs/2602.06625)  
**Date:** 2026-02-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2602.06625v1/

## Key points

- Adaptivity constraints
- Biases influenced by non-semantic cues
- Inconsistencies in evaluation
- Treats judging behavior as a learnable policy
- Constructs high-information-density judging dataset with aligned supervision signals
- Adopts curriculum-style training paradigm to enhance rubric adherence, mitigate biases, and ensure consistency across different evaluation modes
- Not solely attributed to model scale
- Consistently improves in the 2B, 4B, and 8B settings compared to larger models like Qwen2.5-72B and DeepSeek-V3-671B
- FairJudge achieves higher scores across various benchmarks compared to existing judge-oriented models like PandaLM and JudgeLM
- Excels in multimodal evaluations
- Maintains competitive accuracy compared to strong baselines
- Strikes a balance between judgment quality, behavioral consistency, and inference efficiency
- Improves fairness and reproducibility in machine learning model assessment practices without introducing new content generation capabilities or posing significant societal risks

## Summary

, , , , 

In the realm of LLM-as-a-Judge systems, there exist significant limitations such as adaptivity constraints, biases influenced by non-semantic cues, and inconsistencies in evaluation. To combat these issues, a novel approach called FairJudge is proposed. Unlike traditional methods that view the judge as a static evaluator, FairJudge treats judging behavior as a learnable policy. By constructing a high-information-density judging dataset with aligned supervision signals, FairJudge adopts a curriculum-style training paradigm to enhance rubric adherence, mitigate biases, and ensure consistency across different evaluation modes. Importantly, the success of FairJudge cannot be solely attributed to model scale. While larger models like Qwen2.5-72B and DeepSeek-V3-671B show varying judging performance across datasets, FairJudge consistently improves in the 2B, 4B, and 8B settings. This suggests that modeling judging behavior explicitly is more crucial than simply increasing parameter count. Comparative analysis with existing judge-oriented models like PandaLM and JudgeLM demonstrates that FairJudge consistently achieves higher scores across various benchmarks, indicating robust generalization capabilities. Ablation studies highlight the importance of consistency-oriented rewards in learning stable judgment behavior. Furthermore, FairJudge excels in multimodal evaluations by maintaining competitive accuracy compared to strong baselines. The framework strikes a balance between judgment quality, behavioral consistency, and inference efficiency. In conclusion, FairJudge presents a unified framework for LLM-as-a-Judge systems that addresses biases and enhances cross-mode consistency through structured data construction and staged training. Experimental results showcase its reliability in automatic judgments across diverse benchmarks while maintaining strong multimodal generalization and efficient inference capabilities. The impact statement emphasizes the potential of FairJudge to improve fairness and reproducibility in machine learning model assessment practices without introducing new content generation capabilities or posing significant societal risks. Overall, this work contributes positively to promoting responsible use of machine learning systems for more reliable evaluation practices.