# Self-critiquing models for assisting human evaluators

**Authors:** William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike  
**arXiv:** [2206.05802](https://arxiv.org/abs/2206.05802)  
**Date:** 2022-06-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.05802v2/

## Key points

- Topic-based summarization involves capturing non-trivial aspects of a text
- Dataset of over 6,000 distinct topical queries and summaries collected from various sources
- Training models using demonstrations for critiqueability, critique, refinement, and helpfulness tasks
- Larger language models are more effective at writing natural language critiques and identifying flaws in summaries
- Models can detect intentional flaws introduced by humans to mislead readers
- Larger models demonstrate better self-critiquing abilities and can integrate their own self-critiques as feedback
- Framework for comparing critiquing ability with generation and discrimination ability introduced
- Large models may possess relevant knowledge that they cannot or do not articulate as critiques
- AI-assisted human feedback can scale the supervision of machine learning systems in evaluating challenging tasks
- Access provided to training datasets and samples from critique assistance experiments for further research purposes.

## Summary

In this study, we focus on the task of topic-based summarization which involves generating summaries that capture non-trivial aspects of a text rather than simply summarizing the entire text. We collected a dataset of over 6,000 distinct topical queries and summaries from various sources such as short stories, Wikipedia articles and web articles. The summaries are typically up to a paragraph long and aim to capture non-trivial aspects of the text. To train our models, we collect demonstrations for different tasks including critiqueability, critique, refinement and helpfulness tasks. We use both model-generated and human-generated inputs for these tasks. After each round of data collection we train a model jointly on all the task demonstrations collected so far. We find that larger language models are more effective at writing natural language critiques and can help humans identify flaws in summaries that they might have missed. These models are able to detect flaws in both model-generated and human-written summaries including intentional flaws introduced by humans to mislead readers. Furthermore, larger models demonstrate better self-critiquing abilities despite producing outputs that are harder to critique. They can also integrate their own self-critiques as feedback to improve the quality of their own summaries. We introduce a framework for comparing critiquing ability with generation and discrimination ability. Our measurements indicate that even large models may possess relevant knowledge that they cannot or do not articulate as critiques. Overall, our findings demonstrate the potential of using AI-assisted human feedback to scale the supervision of machine learning systems in evaluating tasks that are challenging for humans to assess directly. We provide access to our training datasets and samples from our critique assistance experiments for further research purposes.