Scaling Laws for Reward Model Overoptimization

In reinforcement learning from human feedback, optimizing against a reward model trained to predict human preferences can lead to overoptimization and hinder the performance of the ground truth. To address this challenge, the authors of this study use a synthetic setup where a fixed "gold-standard" reward model acts as a substitute for humans and provides labels for training a proxy reward model. The study investigates how the gold reward model score changes when optimizing against the proxy reward model using two different methods: reinforcement learning and best-of-$n$ sampling. The authors find that the relationship between these scores follows a different functional form depending on the optimization method used. Additionally, they observe that the coefficients in this relationship scale smoothly with the number of reward model parameters. Various factors such as size of the reward model dataset, number of reward model and policy parameters, and coefficient of KL penalty added to the reward in reinforcement learning are explored to understand their impact on this relationship. By analyzing these empirical results, implications for theoretical considerations in AI alignment are discussed. Overall, this study provides insights into how optimizing against imperfect proxy models can affect performance in reinforcement learning from human feedback scenarios. The findings highlight important considerations for balancing optimization against such models while maintaining alignment with ground truth objectives.

Reinforcement Learning From Human Feedback: Optimizing Against a Proxy Model

Humans have long been the primary source of feedback for reinforcement learning (RL) agents. However, optimizing against a reward model trained to predict human preferences can lead to overoptimization and hinder the performance of the ground truth. To address this challenge, researchers have proposed using a fixed "gold-standard" reward model as a substitute for humans and providing labels for training a proxy reward model. In this article, we will discuss how optimizing against such models affects performance in RL from human feedback scenarios.

Background

In recent years, there has been an increasing interest in developing methods that allow RL agents to learn from human feedback. This is due to the fact that humans are able to provide more accurate and reliable rewards than automated systems. However, one major challenge with using human feedback is that it can be difficult to optimize against due to its subjective nature. As such, many researchers have proposed using proxy models as substitutes for humans when training RL agents.

Study Overview

The authors of this study use a synthetic setup where they employ a gold-standard reward model as their ground truth objective and train a proxy reward model on labels provided by the gold-standard model. The goal of their research was twofold: firstly, they wanted to investigate how the gold reward model score changes when optimizing against the proxy reward model; secondly, they wanted to explore various factors such as size of the dataset used for training both models and number of parameters in each respective model which could affect this relationship between scores.

To achieve these goals, two different optimization methods were employed: reinforcement learning (RL) and best-of-$n$ sampling (BOS). For each method, various experiments were conducted where different values were assigned for factors such as size of datasets used for training both models or number of parameters in each respective model etc., so that their impact on score differences could be analyzed empirically.

Findings

The authors found that depending on which optimization method was used - RL or BOS -the relationship between scores followed different functional forms; additionally they observed that coefficients in this relationship scaled smoothly with respect to number of parameters present in either type of models i.e., gold standard or proxy ones respectively . Furthermore , other factors like size of datasets used while training both types , coefficient added during KL penalty etc also had an effect on these relationships .

Implications & Conclusion

By analyzing these empirical results , implications related theoretical considerations about AI alignment can be discussed . Overall , findings from this study provide insights into how optimizing against imperfect proxy models can affect performance in reinforcement learning from human feedback scenarios . It highlights important considerations regarding balancing optimization against such proxies while maintaining alignment with ground truth objectives .