Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Same question Not relevant

   Scaling Laws for Reward Model Overoptimization

   Leo Gao, John Schulman, Jacob Hilton · 2022 · Machine Learning · cited 1,475 times · AI summary available

   Scaling Laws measures how optimizing a learned preference reward changes ground-truth performance, the same reward-overoptimization problem studied here.

   Abstract

   In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learni…

2. Close Broader context Not relevant

   Robust Synthetic Control

   Muhammad Jehangir Amjad, Devavrat Shah, Dennis Shen · 2017 · Econometrics · cited 172 times · AI summary available

   Robust Synthetic Control estimates counterfactual outcomes in comparative studies, rather than controlling agents with preference-learned rewards.

   Abstract

   We present a robust generalization of the synthetic control method for comparative case studies. Like the classical method, we present an algorithm to estimate the unobservable counterfactual of a treatment unit. A distinguishing feature of our algorithm is that of de-noising the data matrix via singular value thresholding, which renders our approach robust in multiple facets: it automatically identifies a good subset of donors, overcomes the challenges of missing data, and continues to work well in settings where covariate information may not be provided. To begin, we establish the condition under which the fundamental assumption in synthetic control-like approaches holds, i.e. when the lin…

3. Close Broader context Not relevant

   Direct Preference Optimization: Your Language Model is Secretly a Reward Model

   Rafael Rafailov, Archit Sharma, Eric Mitchell et al. · 2023 · Machine Learning · cited 11,573 times · AI summary available

   Direct Preference Optimization aligns language models with human preferences, while this paper studies how planners exploit preference-learned rewards.

   Abstract

   While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward…

4. Close Broader context Not relevant

   Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

   Alexander Pan, Chan Jun Shern, Andy Zou et al. · 2023 · Machine Learning · cited 267 times · AI summary available

   MACHIAVELLI evaluates trade-offs between reward and harmful behavior in agents, rather than reward-model exploitation in robot control.

   Abstract

   Artificial agents have traditionally been trained to maximize reward, which may incentivize power-seeking and deception, analogous to how next-token prediction in language models (LMs) may incentivize toxicity. So do agents naturally learn to be Machiavellian? And how do we measure these behaviors in general-purpose models such as GPT-4? Towards answering these questions, we introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios that center on social decision-making. Scenario labeling is automated with LMs, which are more performant than human annotators. We mathematize dozens of harmful behaviors and use our annotatio…

5. Close Broader context Not relevant

   Explore or Confirm? LLM Agents Misallocate Measurements Under Noise

   Abdul Karim, Matt Keon · 2026 · Artificial Intelligence · AI summary available

   Explore or Confirm studies language agents allocating experiments under noisy measurements, not planners optimizing learned preference rewards.

   In short

   In simulated noisy assay campaigns, five LLM agents misallocate measurement budgets by over-confirming or under-confirming, while candidate support helps some over-confirmers.

6. Close Broader context Not relevant

   The Leaderboard Illusion

   Shivalika Singh, Yiyang Nan, Alex Wang et al. · 2025 · Artificial Intelligence · cited 85 times · AI summary available

   The Leaderboard Illusion examines selective disclosure distorting benchmark scores, rather than optimization exploiting an imperfect reward model.

   Abstract

   Measuring progress is fundamental to the advancement of any scientific field. As benchmarks play an increasingly central role, they also grow more susceptible to distortion. Chatbot Arena has emerged as the go-to leaderboard for ranking the most capable AI systems. Yet, in this work we identify systematic issues that have resulted in a distorted playing field. We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired. We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results. At an ex…

7. Close Broader context Not relevant

   Statistical Rejection Sampling Improves Preference Optimization

   Tianqi Liu, Yao Zhao, Rishabh Joshi et al. · 2023 · Computation and Language · cited 385 times · AI summary available

   Statistical Rejection Sampling improves language-model preference optimization, whereas this paper evaluates sampling-based robot control against learned rewards.

   Abstract

   Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as attractive alternatives, offering improvements in stability and scalability while maintaining competitive performance. SLiC refines its loss function using sequence pairs sampled from a supervised fine-tuned (SFT) policy, while DPO directly optimizes language models based on preference data, foregoing the…

8. Close Broader context Not relevant

   Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization

   Soham Sane · 2025 · Machine Learning · preprint, not peer reviewed · cited 28 times · AI summary available

   Hybrid GRPO uses multi-sample action evaluation for policy optimization, while this paper tests sampling-based planners optimizing learned preference rewards.

   Abstract

   Hybrid Group Relative Policy Optimization (Hybrid GRPO) is a reinforcement learning framework that extends Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) by incorporating empirical multi-sample action evaluation while preserving the stability of value function-based learning. Unlike DeepSeek GRPO, which eliminates the value function in favor of purely empirical reward estimation, Hybrid GRPO introduces a structured advantage computation method that balances empirical action sampling with bootstrapped value estimation. This approach enhances sample efficiency, improves learning stability, and mitigates variance amplification observed in purely empirical metho…