Large Language Models as Optimizers

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   Large Language Models Are Human-Level Prompt Engineers

   Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han et al. · 2022 · Machine Learning · AI summary available

   Abstract

   By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality o…

2. Very close Not relevant

   Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

   Corby Rosset, Ching-An Cheng, Arindam Mitra et al. · 2024 · Machine Learning · AI summary available

   Abstract

   This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF), which traditionally separates reward learning and subsequent policy optimization. However, such a reward maximization approach is limited by the nature of "point-wise" rewards (such as Bradley-Terry model), which fails to express complex intransitive or cyclic preference relations. While advances on RLHF show reward learning and policy optimization can be merged into a single contrastive objective for stability, they yet still remai…

3. Close Not relevant

   Direct Preference Optimization: Your Language Model is Secretly a Reward Model

   Rafael Rafailov, Archit Sharma, Eric Mitchell et al. · 2023 · Machine Learning · AI summary available

   Abstract

   While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward…

4. Close Not relevant

   Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

   Yuxin Wen, Neel Jain, John Kirchenbauer et al. · 2023 · Machine Learning · AI summary available

   Abstract

   The strength of modern generative models lies in their ability to be controlled through text-based prompts. Typical "hard" prompts are made from interpretable words and tokens, and must be hand-crafted by humans. There are also "soft" prompts, which consist of continuous feature vectors. These can be discovered using powerful optimization methods, but they cannot be easily interpreted, re-used across models, or plugged into a text-based interface. We describe an approach to robustly optimize hard text prompts through efficient gradient-based optimization. Our approach automatically generates hard text-based prompts for both text-to-image and text-to-text applications. In the text-to-image se…

5. Close Not relevant

   Reward Design with Language Models

   Minae Kwon, Sang Michael Xie, Kalesha Bullard et al. · 2023 · Machine Learning · AI summary available

   Abstract

   Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of the desired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning of training. During training, the LLM evalu…

6. Close Not relevant

   Efficient Exploration for LLMs

   Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao et al. · 2024 · Machine Learning · AI summary available

   Abstract

   We present evidence of substantial benefit from efficient exploration in gathering human feedback to improve large language models. In our experiments, an agent sequentially generates queries while fitting a reward model to the feedback received. Our best-performing agent generates queries using double Thompson sampling, with uncertainty represented by an epistemic neural network. Our results demonstrate that efficient exploration enables high levels of performance with far fewer queries. Further, both uncertainty estimation and the choice of exploration scheme play critical roles.

7. Close Not relevant

   Offline Reinforcement Learning for LLM Multi-Step Reasoning

   Huaijie Wang, Shibo Hao, Hanze Dong et al. · 2024 · Machine Learning · AI summary available

   Abstract

   Improving the multi-step reasoning ability of large language models (LLMs) with offline reinforcement learning (RL) is essential for quickly adapting them to complex tasks. While Direct Preference Optimization (DPO) has shown promise in aligning LLMs with human preferences, it is less suitable for multi-step reasoning tasks because (1) DPO relies on paired preference data, which is not readily available for multi-step reasoning tasks, and (2) it treats all tokens uniformly, making it ineffective for credit assignment in multi-step reasoning tasks, which often come with sparse reward. In this work, we propose OREO (Offline Reasoning Optimization), an offline RL method for enhancing LLM multi-…

8. Close Not relevant

   KTO: Model Alignment as Prospect Theoretic Optimization

   Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff et al. · 2024 · Machine Learning · AI summary available

   Abstract

   Kahneman & Tversky's $\textit{prospect theory}$ tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases -- the success of these objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed to them belonging to a family of loss functions that we call $\textit{human-aware losses}$ (HALOs). However, the utility functions these methods attribute to humans still differ from those in the prospect theory literature. Using a Kahneman-Tversky model of human utility, we propose a HALO that directly ma…