Self-critiquing models for assisting human evaluators
AI-generated Key Points
- The authors fine-tuned large language models to write natural language critiques using behavioral cloning.
- The main task used for evaluation was topic-based summarization, where the summary focuses on a specific aspect of a text rather than trying to summarize the whole text.
- The authors collected their own dataset of over 6,000 distinct topical queries and summaries, on over 2,000 distinct passages.
- Larger models write more helpful critiques and are better at self-critiquing despite having harder-to-critique outputs.
- A framework for comparing critiquing ability to generation and discrimination ability was introduced.
- Even large models may still have relevant knowledge they cannot or do not articulate as critiques.
- AI-assisted human feedback can be used to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly.
- The authors released their training datasets as well as samples from their critique assistance experiments.
Authors: William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike
Abstract: We fine-tune large language models to write natural language critiques (natural language critical comments) using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Our models help find naturally occurring flaws in both model and human written summaries, and intentional flaws in summaries written by humans to be deliberately misleading. We study scaling properties of critiquing with both topic-based summarization and synthetic tasks. Larger models write more helpful critiques, and on most tasks, are better at self-critiquing, despite having harder-to-critique outputs. Larger models can also integrate their own self-critiques as feedback, refining their own summaries into better ones. Finally, we motivate and introduce a framework for comparing critiquing ability to generation and discrimination ability. Our measurements suggest that even large models may still have relevant knowledge they cannot or do not articulate as critiques. These results are a proof of concept for using AI-assisted human feedback to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly. We release our training datasets, as well as samples from our critique assistance experiments.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Similar papers summarized with our AI tools
Navigate through even more similar papers through atree representation
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.