# TravelPlanner: A Benchmark for Real-World Planning with Language Agents

**Authors:** Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, Yu Su  
**arXiv:** [2402.01622](https://arxiv.org/abs/2402.01622)  
**Date:** 2024-02-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.01622v1/

## Key points

- Planning is a fundamental aspect in the pursuit of artificial intelligence.
- Recent advancements in language agents powered by large language models (LLMs) have shown capabilities in tool usage and reasoning.
- A new benchmark called TravelPlanner has been introduced to assess the ability of language agents in complex planning tasks, focusing on travel planning as a real-world scenario.
- Despite promising potential, current language models struggle with handling complex planning tasks, with even advanced models like GPT-4 achieving a low success rate in the benchmark.
- Language agents face challenges in maintaining focus, utilizing appropriate tools for information gathering, and managing multiple constraints simultaneously.
- The study "TravelPlanner: A Benchmark for Real-World Planning with Language Agents" provides insights into the limitations and potentials of current language models when applied to complex planning tasks.

## Summary

In the pursuit of artificial intelligence, planning has always been a fundamental aspect. Earlier AI agents primarily operated in constrained settings due to the lack of cognitive substrates required for human-level planning. However, recent advancements in language agents powered by large language models (LLMs) have showcased intriguing capabilities such as tool usage and reasoning. The question arises: can these language agents effectively engage in planning tasks in more intricate scenarios? These scenarios were previously beyond the reach of traditional AI agents. To address this query, a new benchmark called TravelPlanner has been introduced. It focuses specifically on travel planning as a common real-world scenario. This benchmark offers a comprehensive sandbox environment with various tools to access nearly four million data records. It also includes 1,225 meticulously curated planning intents and reference plans to facilitate evaluations. Despite the promising potential of language agents, comprehensive assessments reveal that current models still struggle with handling complex planning tasks. Even GPT-4, one of the most advanced language models available, only achieves a success rate of 0.6% in this challenging benchmark. Language agents face difficulties in maintaining focus on the task at hand and utilizing appropriate tools for information gathering while managing multiple constraints simultaneously. Nevertheless, it is crucial to acknowledge that the mere ability of language agents to tackle such intricate problems represents significant progress in the field of artificial intelligence. While TravelPlanner presents a formidable challenge for existing language agents, it also serves as a valuable testbed for future advancements in this domain. The study "TravelPlanner: A Benchmark for Real-World Planning with Language Agents" authored by Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao and Yu Su provides insights into the limitations and potentials of current language models when applied to complex planning tasks. This work signifies an ongoing effort towards enhancing the capabilities of AI systems for real-world applications requiring sophisticated planning abilities.