# Chain of Hindsight Aligns Language Models with Feedback

**Authors:** Hao Liu, Carmelo Sferrazza, Pieter Abbeel  
**arXiv:** [2302.02676](https://arxiv.org/abs/2302.02676)  
**Date:** 2023-02-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.02676v6/

## Key points

- Authors emphasize the importance of learning from human preferences in language models
- Previous methods have limitations in data utilization and optimization
- The proposed technique is called Chain of Hindsight
- Feedback is converted into sentences to fine-tune the language model
- Model learns to identify and correct negative attributes or errors based on feedback
- Significant improvements observed in aligning models with human preferences, particularly in summarization and dialogue tasks
- Positive evaluations received in human assessments
- Chain of Hindsight offers an easy-to-optimize approach for learning from any type of feedback

## Summary

In their paper titled "Chain of Hindsight Aligns Language Models with Feedback," authors Hao Liu, Carmelo Sferrazza, and Pieter Abbeel emphasize the importance of learning from human preferences in language models. They highlight that previous methods either rely on hand-picked model generations favored by human annotators, limiting data utilization and general applicability, or depend on imperfect reward functions and reinforcement learning which are challenging to optimize. To address these limitations, the authors propose a novel technique called Chain of Hindsight. Inspired by how humans learn from extensive feedback presented in languages, they convert all types of feedback into sentences. These sentences are then used to fine-tune the language model leveraging its language comprehension capabilities. The model is conditioned on a sequence of model generations paired with feedback enabling it to generate outputs based on feedback and learn to identify and correct negative attributes or errors. The authors applied their method to large language models and observed significant improvements in aligning the models with human preferences compared to previous approaches. Specifically, they noted substantial enhancements in summarization and dialogue tasks. Moreover, their approach received markedly positive evaluations in human assessments. Overall, this research highlights the effectiveness of the Chain of Hindsight technique in training language models to align with human preferences. By converting various forms of feedback into sentences and conditioning the model on generation-feedback pairs, this method offers an easy-to-optimize approach for learning from any type of feedback regardless of its polarity.