Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Proche Même question Pas pertinent

   Scaling Laws for Reward Model Overoptimization

   Leo Gao, John Schulman, Jacob Hilton · 2022 · Machine Learning · cité 1 475 fois · Résumé IA disponible

   Scaling Laws for Reward Model Overoptimization mesure comment l’optimisation d’une récompense proxy entraînée sur des préférences affecte la performance réelle.

   Résumé scientifique

   In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learni…

2. Proche Contexte plus large Pas pertinent

   Robust Synthetic Control

   Muhammad Jehangir Amjad, Devavrat Shah, Dennis Shen · 2017 · Econometrics · cité 172 fois · Résumé IA disponible

   Robust Synthetic Control estime des contrefactuels dans des études comparatives, sans traiter l’optimisation de récompenses apprises.

   Résumé scientifique

   We present a robust generalization of the synthetic control method for comparative case studies. Like the classical method, we present an algorithm to estimate the unobservable counterfactual of a treatment unit. A distinguishing feature of our algorithm is that of de-noising the data matrix via singular value thresholding, which renders our approach robust in multiple facets: it automatically identifies a good subset of donors, overcomes the challenges of missing data, and continues to work well in settings where covariate information may not be provided. To begin, we establish the condition under which the fundamental assumption in synthetic control-like approaches holds, i.e. when the lin…

3. Proche Contexte plus large Pas pertinent

   Direct Preference Optimization: Your Language Model is Secretly a Reward Model

   Rafael Rafailov, Archit Sharma, Eric Mitchell et al. · 2023 · Machine Learning · cité 11 573 fois · Résumé IA disponible

   Direct Preference Optimization aligne des modèles de langage sur des préférences humaines, plutôt que de planifier des actions contre une récompense apprise.

   Résumé scientifique

   While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward…

4. Proche Contexte plus large Pas pertinent

   Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

   Alexander Pan, Chan Jun Shern, Andy Zou et al. · 2023 · Machine Learning · cité 267 fois · Résumé IA disponible

   MACHIAVELLI étudie les compromis entre récompense et comportements éthiques d’agents dans des jeux narratifs.

   Résumé scientifique

   Artificial agents have traditionally been trained to maximize reward, which may incentivize power-seeking and deception, analogous to how next-token prediction in language models (LMs) may incentivize toxicity. So do agents naturally learn to be Machiavellian? And how do we measure these behaviors in general-purpose models such as GPT-4? Towards answering these questions, we introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios that center on social decision-making. Scenario labeling is automated with LMs, which are more performant than human annotators. We mathematize dozens of harmful behaviors and use our annotatio…

5. Proche Contexte plus large Pas pertinent

   Explore or Confirm? LLM Agents Misallocate Measurements Under Noise

   Abdul Karim, Matt Keon · 2026 · Artificial Intelligence · Résumé IA disponible

   Explore or Confirm? mesure comment des agents LLM répartissent des expériences sous bruit, sans étudier la recherche contre une récompense de préférences.

   Résumé scientifique

   As model reasoning gets cheaper, the physical experiment becomes the slow and costly step in discovery. Each experiment returns a noisy reading: a high value can mean a good candidate or a lucky draw. With a fixed budget, an AI agent must keep choosing between exploring a new candidate and confirming a promising one by re-testing it. We measure that choice in Wellocate, a simulated assay campaign with known true values. It is built on a generated landscape and on two real protein datasets, GB1 and PhoQ, scrambled so no model can answer from memory. We test five language models as minimal agents, with no tools and no memory beyond the history in each prompt. They misallocate in both direction…

6. Proche Contexte plus large Pas pertinent

   The Leaderboard Illusion

   Shivalika Singh, Yiyang Nan, Alex Wang et al. · 2025 · Artificial Intelligence · cité 85 fois · Résumé IA disponible

   The Leaderboard Illusion analyse le biais de scores causé par la sélection des résultats publiés, et non par un planificateur exploitant une récompense.

   Résumé scientifique

   Measuring progress is fundamental to the advancement of any scientific field. As benchmarks play an increasingly central role, they also grow more susceptible to distortion. Chatbot Arena has emerged as the go-to leaderboard for ranking the most capable AI systems. Yet, in this work we identify systematic issues that have resulted in a distorted playing field. We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired. We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results. At an ex…

7. Proche Contexte plus large Pas pertinent

   Statistical Rejection Sampling Improves Preference Optimization

   Tianqi Liu, Yao Zhao, Rishabh Joshi et al. · 2023 · Computation and Language · cité 385 fois · Résumé IA disponible

   Statistical Rejection Sampling améliore l’optimisation de préférences pour les modèles de langage, au lieu d’évaluer le contrôle planifié.

   Résumé scientifique

   Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as attractive alternatives, offering improvements in stability and scalability while maintaining competitive performance. SLiC refines its loss function using sequence pairs sampled from a supervised fine-tuned (SFT) policy, while DPO directly optimizes language models based on preference data, foregoing the…

8. Proche Autre approche du problème Pas pertinent

   Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization

   Soham Sane · 2025 · Machine Learning · preprint, non relu par les pairs · cité 28 fois · Résumé IA disponible

   Hybrid GRPO évalue plusieurs actions pour optimiser une politique par apprentissage par renforcement, contrairement à MPPI et best-of-N testés ici.

   Résumé scientifique

   Hybrid Group Relative Policy Optimization (Hybrid GRPO) is a reinforcement learning framework that extends Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) by incorporating empirical multi-sample action evaluation while preserving the stability of value function-based learning. Unlike DeepSeek GRPO, which eliminates the value function in favor of purely empirical reward estimation, Hybrid GRPO introduces a structured advantage computation method that balances empirical action sampling with bootstrapped value estimation. This approach enhances sample efficiency, improves learning stability, and mitigates variance amplification observed in purely empirical metho…