# Self-Taught Evaluators

**Authors:** Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, Xian Li  
**arXiv:** [2408.02666](https://arxiv.org/abs/2408.02666)  
**Date:** 2024-08-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.02666v1/

## Key points

- Model-based evaluation is crucial for successful model development
- Traditional training evaluators involve collecting human preference judgments, which can be costly and result in stale data
- An innovative approach using synthetic training data exclusively is introduced to enhance evaluators without relying on human annotations
- The Self-Taught Evaluator achieves significant improvement in performance without any labeled preference data
- Combining synthetic preference data with human-labeled preference data shows strong performance across different mixing weights
- Analysis of instruction complexity reveals differences between curated datasets and full datasets before instruction selection
- The scalable approach offers a robust solution for constructing a generalist evaluator to assess LLM outputs through synthetic preferences without human annotation reliance

## Summary

Model-based evaluation is crucial for successful model development. It serves as a reward model for training and a substitute for human evaluation. Traditionally, training evaluators involves collecting a large amount of human preference judgments over model responses. However, this can be costly and result in stale data as models improve. In an effort to enhance evaluators without relying on human annotations, this work introduces an innovative approach that utilizes synthetic training data exclusively. The process begins with unlabeled instructions and implements an iterative self-improvement scheme that generates contrasting model outputs. An LLM-as-a-Judge is then trained to provide reasoning traces and final judgments based on these outputs. The training process is repeated at each iteration using the improved predictions. Remarkably, the Self-Taught Evaluator achieves significant improvement in performance without any labeled preference data. It enhances a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (or 88.7 with majority vote) on RewardBench. This advancement surpasses the capabilities of commonly used LLM judges like GPT-4 and matches the performance of top-performing reward models trained with labeled examples. Additionally, the study explores the impact of combining synthetic preference data with human-labeled preference data on model performance. By merging synthetic preferences generated from WildChat prompts with human-labeled HelpSteer2 dataset, the models demonstrate strong performance across different mixing weights. Furthermore, an analysis of instruction complexity reveals insights into the length distribution of curated training sets compared to full datasets before instruction selection. The curated dataset showcases more complex instructions involving logical reasoning/science while the full dataset leans towards relationships and entertainment content. In conclusion, this scalable approach offers a robust solution for constructing a generalist evaluator to assess LLM outputs through synthetic preferences without human annotation reliance. The Self-Taught Evaluator's iterative training over synthetic preferences significantly enhances the accuracy levels of a strong seed LLM (Llama3-70B-Instruct), setting new benchmarks in generative LLM-as-a-Judge methods within the field of model-based evaluation research.