Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents

Authors: Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, Rafael Rafailov

Submitted to arXiv on 13 Aug. 2024
Abstract

Large Language Models (LLMs) have shown remarkable capabilities in natural language tasks requiring complex reasoning, yet their application in agentic, multi-step reasoning within interactive environments remains a difficult challenge. Traditional supervised pre-training on static datasets falls short in enabling autonomous agent capabilities needed to perform complex decision-making in dynamic settings like web navigation. Previous attempts to bridge this ga-through supervised fine-tuning on curated expert demonstrations-often suffer from compounding errors and limited exploration data, resulting in sub-optimal policy outcomes. To overcome these challenges, we propose a framework that combines guided Monte Carlo Tree Search (MCTS) search with a self-critique mechanism and iterative fine-tuning on agent interactions using an off-policy variant of the Direct Preference Optimization (DPO) algorithm. Our method allows LLM agents to learn effectively from both successful and unsuccessful trajectories, thereby improving their generalization in complex, multi-step reasoning tasks. We validate our approach in the WebShop environment-a simulated e-commerce platform where it consistently outperforms behavior cloning and reinforced fine-tuning baseline, and beats average human performance when equipped with the capability to do online search. In real-world booking scenarios, our methodology boosts Llama-3 70B model's zero-shot performance from 18.6% to 81.7% success rate (a 340% relative increase) after a single day of data collection and further to 95.4% with online search. We believe this represents a substantial leap forward in the capabilities of autonomous agents, paving the way for more sophisticated and reliable decision-making in real-world settings.

The papers around it, by date and closeness

Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper

How to read this map? Illustrated guide

Closest papers Prior work it builds on Papers that cite it

Papers around this one

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close Cites this paper
    AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents
    Ke Yang, Yao Liu, Sapana Chaudhary et al. · 2024 · Artificial Intelligence · cited 147 times · AI summary available
    Abstract

    Autonomy via agents using large language models (LLMs) for personalized, standardized tasks boosts human efficiency. Automating web tasks (like booking hotels within a budget) is increasingly sought after. Fulfilling practical needs, the web agent also serves as an important proof-of-concept example for various agent grounding scenarios, with its success promising advancements in many future applications. Prior research often handcrafts web agent strategies (e.g., prompting templates, multi-agent systems, search methods, etc.) and the corresponding in-context examples, which may not generalize well across all real-world scenarios. On the other hand, there has been limited study on the misali…

  2. Very close
    SimuRA: Towards General Goal-Oriented Agent via Simulative Reasoning Architecture with LLM-Based World Model
    Mingkai Deng, Jinyu Hou, Yilin Shen et al. · 2025 · Artificial Intelligence · cited 8 times · AI summary available
    Abstract

    AI agents built on large language models (LLMs) hold enormous promise, but current practice focuses on a one-task-one-agent approach, which not only falls short of scalability and generality, but also suffers from the fundamental limitations of autoregressive LLMs. On the other hand, humans are general agents who reason by mentally simulating the outcomes of their actions and plans. Moving towards a more general and powerful AI agent, we introduce SimuRA, a goal-oriented architecture for generalized agentic reasoning. Based on a principled formulation of optimal agent in any environment, \modelname overcomes the limitations of autoregressive reasoning by introducing a world model for plannin…

  3. Very close
    Tree Search for Language Model Agents
    Jing Yu Koh, Stephen McAleer, Daniel Fried et al. · 2024 · Artificial Intelligence · cited 165 times · AI summary available
    Abstract

    Autonomous agents powered by language models (LMs) have demonstrated promise in their ability to perform decision-making tasks such as web automation. However, a key limitation remains: LMs, primarily optimized for natural language understanding and generation, struggle with multi-step reasoning, planning, and using environmental feedback when attempting to solve realistic computer tasks. Towards addressing this, we propose an inference-time search algorithm for LM agents to explicitly perform exploration and multi-step planning in interactive web environments. Our approach is a form of best-first tree search that operates within the actual environment space, and is complementary with most e…

  4. Very close
    ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning
    Xiao Yu, Baolin Peng, Vineeth Vajipey et al. · 2024 · International Conference on Learning Representations · cited 68 times · Not summarized yet
    Abstract

    Autonomous agents have demonstrated significant potential in automating complex multistep decision-making tasks. However, even state-of-the-art vision-language models (VLMs), such as GPT-4o, still fall short of human-level performance, particularly in intricate web environments and long-horizon tasks. To address these limitations, we present ExACT, an approach to combine test-time search and self-learning to build o1-like models for agentic applications. We first introduce Reflective Monte Carlo Tree Search (R-MCTS), a novel test time algorithm designed to enhance AI agents' ability to explore decision space on the fly. R-MCTS extends traditional MCTS by 1) incorporating contrastive reflecti…

  5. Very close
    Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency
    Zhihan Liu, Hao Hu, Shenao Zhang et al. · 2023 · Artificial Intelligence · preprint, not peer reviewed · cited 57 times · AI summary available
    Abstract

    Large language models (LLMs) demonstrate impressive reasoning abilities, but translating reasoning into actions in the real world remains challenging. In particular, it remains unclear how to complete a given task provably within a minimum number of interactions with the external environment, e.g., through an internal mechanism of reasoning. To this end, we propose a principled framework with provable regret guarantees to orchestrate reasoning and acting, which we call "reason for future, act for now" (\texttt{RAFA}). Specifically, we design a prompt template for reasoning that learns from the memory buffer and plans a future trajectory over a long horizon ("reason for future"). At each step…

  6. Very close
    Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman et al. · 2023 · Artificial Intelligence · cited 749 times · AI summary available
    Abstract

    While large language models (LLMs) have demonstrated impressive performance on a range of decision-making tasks, they rely on simple acting processes and fall short of broad deployment as autonomous agents. We introduce LATS (Language Agent Tree Search), a general framework that synergizes the capabilities of LLMs in planning, acting, and reasoning. Drawing inspiration from Monte Carlo tree search in model-based reinforcement learning, LATS employs LLMs as agents, value functions, and optimizers, repurposing their latent strengths for enhanced decision-making. What is crucial in this method is the use of an environment for external feedback, which offers a more deliberate and adaptive proble…

  7. Very close
    SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
    Xuan-Phi Nguyen, Shrey Pandit, Revanth Gangi Reddy et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 30 times · Not summarized yet
    Abstract

    Equipping large language models (LLMs) with complex, interleaved reasoning and tool-use capabilities has become a key focus in agentic AI research, especially with recent advances in reasoning-oriented (``thinking'') models. Such capabilities are key to unlocking a number of important applications. One such application is Deep Research (DR), which requires extensive search and reasoning over many sources. Our work in this paper focuses on the development of native Autonomous Single-Agent models for DR featuring minimal web crawling and Python tool integration. Unlike multi-agent systems, where agents take up pre-defined roles and are told what to do at each step in a static workflow, an auto…

  8. Very close
    AgentSquare: Automatic LLM Agent Search in Modular Design Space
    Yu Shang, Yu Li, Keyu Zhao et al. · 2024 · Computation and Language · cited 188 times · Not summarized yet
    Abstract

    Recent advancements in Large Language Models (LLMs) have led to a rapid growth of agentic systems capable of handling a wide range of complex tasks. However, current research largely relies on manual, task-specific design, limiting their adaptability to novel tasks. In this paper, we introduce a new research problem: Modularized LLM Agent Search (MoLAS). We propose a modular design space that abstracts existing LLM agent designs into four fundamental modules with uniform IO interface: Planning, Reasoning, Tool Use, and Memory. Building on this design space, we present a novel LLM agent search framework called AgentSquare, which introduces two core mechanisms, i.e., module evolution and recom…

Prior work it builds on

  1. Close
    A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
    Izzeddin Gur, Hiroki Furuta, Austin L. Huang et al. · 2023 · International Conference on Learning Representations · cited 433 times · Not summarized yet
    Abstract

    Pre-trained large language models (LLMs) have recently achieved better generalization and sample efficiency in autonomous web automation. However, the performance on real-world websites has still suffered from (1) open domainness, (2) limited context length, and (3) lack of inductive bias on HTML. We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions. WebAgent plans ahead by decomposing instructions into canonical sub-instructions, summarizes long HTML documents into task-relevant snippets, and acts on websites via Python programs generated from those. We design WebAgent with Flan-U-PaLM, for gro…

  2. Close
    ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
    Yifei Zhou, Andrea Zanette, Jiayi Pan et al. · 2024 · International Conference on Machine Learning · cited 275 times · Not summarized yet
    Abstract

    A broad use case of large language models (LLMs) is in goal-directed decision-making tasks (or"agent"tasks), where an LLM needs to not just generate completions for a given prompt, but rather make intelligent decisions over a multi-turn interaction to accomplish a task (e.g., when interacting with the web, using tools, or providing customer support). Reinforcement learning (RL) provides a general paradigm to address such agent tasks, but current RL methods for LLMs largely focus on optimizing single-turn rewards. By construction, most single-turn RL methods cannot endow LLMs with the ability to intelligently seek information over multiple turns, perform credit assignment, or reason about the…

  3. Close Used in depth by this paper
    Tree of Thoughts: Deliberate Problem Solving with Large Language Models
    Shunyu Yao, Dian Yu, Jeffrey Zhao et al. · 2023 · Neural Information Processing Systems · cited 5,271 times · Not summarized yet
    Abstract

    Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate d…

  4. Close Used in depth by this paper
    Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
    Xuan Zhang, Chao Du, Tianyu Pang et al. · 2024 · Neural Information Processing Systems · cited 187 times · Not summarized yet
    Abstract

    The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial …

  5. Close
    Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
    Yifan Song, Da Yin, Xiang Yue et al. · 2024 · Annual Meeting of the Association for Computational Linguistics · cited 227 times · Not summarized yet
    Abstract

    Large Language Models (LLMs) have become integral components in various autonomous agent systems. In this study, we present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Contrary to previous studies that exclusively train on successful expert trajectories, our method allows agents to learn from their exploration failures. This leads to improved performance through an iterative optimization framework. During the exploration phase, the agent interacts with the environment while completing given tasks, gathering failure trajectories to create contrastive trajectory pairs. In the subseque…

Papers that cite it

  1. Very close
    WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
    Zehan Qi, Xiao Liu, Iat Long Iong et al. · 2024 · Computation and Language · preprint, not peer reviewed · cited 305 times · AI summary available
    Abstract

    Large language models (LLMs) have shown remarkable potential as autonomous agents, particularly in web-based tasks. However, existing LLM web agents heavily rely on expensive proprietary LLM APIs, while open LLMs lack the necessary decision-making capabilities. This paper introduces WebRL, a self-evolving online curriculum reinforcement learning framework designed to train high-performance web agents using open LLMs. WebRL addresses three key challenges in building LLM web agents, including the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that generates new tasks from unsu…

  2. Very close
    Scaling Autonomous Agents via Automatic Reward Modeling And Planning
    Zhenfang Chen, Delin Chen, Rui Sun et al. · 2025 · International Conference on Learning Representations · cited 24 times · Not summarized yet
    Abstract

    Large language models (LLMs) have demonstrated remarkable capabilities across a range of text-generation tasks. However, LLMs still struggle with problems requiring multi-step decision-making and environmental feedback, such as online shopping, scientific reasoning, and mathematical problem-solving. Unlike pure text data, collecting large-scale decision-making data is challenging. Moreover, many powerful LLMs are only accessible through APIs, which hinders their fine-tuning for agent tasks due to cost and complexity. To address LLM agents' limitations, we propose a framework that can automatically learn a reward model from the environment without human annotations. This model can be used to …

  3. Close Uses this paper in depth
    Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training
    Yuan-Zhao Zhai, Huan-Xi Liu, Zhuo Zhang et al. · 2025 · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · cited 4 times · Not summarized yet
    Abstract

    Large Language Model (LLM) agents frequently produce sub-optimal actions when tackling complex, multi-step decision-making tasks. Employing self-critique to identify flaws and suggest enhancements is an effective strategy for refining actions. Although trajectory-level critique is commonly employed, it often fails to identify flawed steps accurately. In this paper, we introduce SLSC-MCTS, a method that integrates Monte Carlo Tree Search with Step-Level Self-Critique to enhance LLM agents during both testing and self-training phases. During decision tree expansion with SLSC-MCTS, the LLM agent initially generates an action, receives environmental feedback, and subsequently generates further a…

  4. Close
    Agentic Reinforced Policy Optimization
    Guan-Ting Dong, Hang-Yu Mao, Kai Ma et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 165 times · Not summarized yet
    Abstract

    Large-scale reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks. In realistic reasoning scenarios, LLMs can often utilize external tools to assist in task-solving processes. However, current RL algorithms inadequately balance the models'intrinsic long-horizon reasoning capabilities and their proficiency in multi-turn tool interactions. To bridge this gap, we propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents. Through preliminary experiments, we observe that LLMs tend to exhibit highl…

  5. Close
    MALT: Improving Reasoning with Multi-Agent LLM Training
    S. Motwani, Chandler Smith, Rocktim Jyoti Das et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 88 times · Not summarized yet
    Abstract

    Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agen…

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 2 November 2026.

The illustrated guide to the papers around a paper