# Critique-out-Loud Reward Models

**Authors:** Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, Prithviraj Ammanabrolu  
**arXiv:** [2408.11791](https://arxiv.org/abs/2408.11791)  
**Date:** 2024-08-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.11791v1/

## Key points

- Traditional reward models in reinforcement learning from human feedback (RLHF) are limited in effectiveness as they make implicit judgments about response quality in a single forward pass through the model.
- Critique-out-Loud (CLoud) reward models address this limitation by generating natural language critiques of assistant responses to explicitly evaluate response quality.
- CLoud models have shown significant improvements in pairwise preference classification accuracy on RewardBench for Llama-3-8B and 70B base models, with increases in accuracy by 4.65 and 5.84 percentage points respectively.
- When used for Best-of-N scoring on ArenaHard, CLoud reward models have led to a Pareto improvement in win rate and offer dynamic inference compute capabilities for self-consistency decoding during reward prediction.
- This study focuses on leveraging critiques to enhance reward model training rather than using oracle critiques or human-labeled critique preferences, distinguishing it from previous research efforts.
- The concept of LLM-as-a-Judge is discussed as a method where large language models evaluate responses based on user-provided grading rubrics, presenting an interesting avenue for future exploration when integrated with CLoud reward models' critique process.
- The innovative CLoud reward models introduced in this study bridge classic reward modeling objectives with LLM-based evaluation approaches like LLM-as-a-Judge, paving the way for more sophisticated and effective preference modeling techniques in RLHF systems.

## Summary

In the realm of reinforcement learning from human feedback (RLHF), traditional reward models are typically trained to predict preference scores without fully utilizing the generation capabilities of large language models (LLMs). This approach limits the effectiveness of reward models as they are required to make implicit judgments about response quality in a single forward pass through the model. To address this limitation and enable reward models to explicitly evaluate response quality, Critique-out-Loud (CLoud) reward models have been introduced. <br>
<br>
CLoud reward models operate by generating a natural language critique of an assistant's response, which is then used to determine a scalar reward for the response quality. In comparison to classic reward models, CLoud models have shown significant improvements in pairwise preference classification accuracy on RewardBench for both Llama-3-8B and 70B base models. Specifically, CLoud models have demonstrated an increase in accuracy by 4.65 and 5.84 percentage points for the 8B and 70B base models respectively.<br>
<br>
Furthermore, when utilized as the scoring model for Best-of-N on ArenaHard, CLoud reward models have led to a Pareto improvement in win rate. Additionally, these models offer dynamic inference compute capabilities that allow for self-consistency decoding during reward prediction.<br>
<br>
Previous research has explored training LLMs to critique responses using oracle critiques or human-labeled critique preferences. However, the approach taken in this work differs by focusing on leveraging critiques to enhance reward model training. While similar studies have demonstrated benefits of conditioning reward scores on critiques, this work stands out by training the reward model to generate its own critiques.<br>
<br>
The concept of LLM-as-a-Judge has also been discussed within this context, where an LLM evaluates responses based on user-provided grading rubrics. While similar to other methods such as Constitutional AI, LLM-as-a-Judge differs in its objective of evaluating responses rather than revising them. The integration of human-crafted grading rubrics from LLM-as-a-Judge with the critique process of CLoud reward models presents an interesting avenue for future exploration.<br>
<br>
In conclusion, this study introduces innovative CLoud reward models that leverage natural language critiques to enhance the training and performance of reinforcement learning from human feedback systems. By bridging classic reward modeling objectives with LLM-based evaluation approaches like LLM-as-a-Judge, this work paves the way for more sophisticated and effective preference modeling techniques in RLHF systems.