# Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents

**Authors:** Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, Rafael Rafailov  
**arXiv:** [2408.07199](https://arxiv.org/abs/2408.07199)  
**Date:** 2024-08-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.07199v1/

## Key points

- Authors propose a novel framework combining guided Monte Carlo Tree Search (MCTS) with self-critique mechanism and iterative fine-tuning on agent interactions using an off-policy variant of the Direct Preference Optimization (DPO) algorithm
- Approach enables LLM agents to learn effectively from successful and unsuccessful trajectories, enhancing generalization abilities in complex decision-making tasks
- Validated methodology in WebShop environment, outperforming behavior cloning and reinforced fine-tuning baselines
- Equipped with online search capabilities, approach surpasses average human performance levels
- Boosts zero-shot performance of Llama-3 70B model from 18.6% to 81.7% success rate after just a single day of data collection; further improves to 95.4% success rate with online search capabilities integrated
- Results represent substantial leap forward in enhancing capabilities of autonomous agents and pave way for more sophisticated decision-making processes in real-world settings

## Summary

In their paper titled "Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents," authors Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov delve into the challenges faced by Large Language Models (LLMs) in agentic, multi-step reasoning within interactive environments. The authors propose a novel framework that combines guided Monte Carlo Tree Search (MCTS) with a self-critique mechanism and iterative fine-tuning on agent interactions using an off-policy variant of the Direct Preference Optimization (DPO) algorithm to overcome these challenges. This approach enables LLM agents to learn effectively from both successful and unsuccessful trajectories, enhancing their generalization abilities in complex decision-making tasks. The authors validate their proposed methodology in the WebShop environment—a simulated e-commerce platform—where it consistently outperforms behavior cloning and reinforced fine-tuning baselines. Furthermore, when equipped with online search capabilities, the approach surpasses average human performance levels. In real-world booking scenarios, the methodology significantly boosts the zero-shot performance of the Llama-3 70B model from 18.6% to 81.7% success rate after just a single day of data collection. With online search capabilities integrated, this success rate further improves to an impressive 95.4%. These results represent a substantial leap forward in enhancing the capabilities of autonomous agents and pave the way for more sophisticated and reliable decision-making processes in real-world settings. By leveraging advanced reasoning techniques and learning mechanisms through , their framework demonstrates significant promise for enabling autonomous AI agents to tackle complex decision-making tasks effectively and efficiently using .