# Self-critiquing models for assisting human evaluators

**Authors:** William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike  
**arXiv:** [2206.05802](https://arxiv.org/abs/2206.05802)  
**Date:** 2022-06-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2206.05802v1/

## Key points

- The authors fine-tuned large language models to write natural language critiques using behavioral cloning.
- The main task used for evaluation was topic-based summarization, where the summary focuses on a specific aspect of a text rather than trying to summarize the whole text.
- The authors collected their own dataset of over 6,000 distinct topical queries and summaries, on over 2,000 distinct passages.
- Larger models write more helpful critiques and are better at self-critiquing despite having harder-to-critique outputs.
- A framework for comparing critiquing ability to generation and discrimination ability was introduced.
- Even large models may still have relevant knowledge they cannot or do not articulate as critiques.
- AI-assisted human feedback can be used to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly.
- The authors released their training datasets as well as samples from their critique assistance experiments.

## Summary

In this study, the authors fine-tuned large language models to write natural language critiques using behavioral cloning. The main task used for evaluation was topic-based summarization, where the summary focuses on a specific aspect of a text rather than trying to summarize the whole text. The authors collected their own dataset of over 6,000 distinct topical queries and summaries, on over 2,000 distinct passages. They trained models jointly on all task demonstrations collected thus far and found that larger models write more helpful critiques and are better at self-critiquing despite having harder-to-critique outputs. They also introduced a framework for comparing critiquing ability to generation and discrimination ability. Their measurements suggest that even large models may still have relevant knowledge they cannot or do not articulate as critiques. These results provide evidence for using AI-assisted human feedback to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly. The authors released their training datasets as well as samples from their critique assistance experiments.