# WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

**Authors:** Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, Dong Yu  
**arXiv:** [2401.13919](https://arxiv.org/abs/2401.13919)  
**Date:** 2024-01-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.13919v1/

## Key points

- Evolution of large language models (LLMs) leading to autonomous applications in real-world scenarios
- Introduction of WebVoyager, a Large Multimodal Model (LMM) powered web agent designed for end-to-end user interaction with actual websites
- Leveraging GPT-4V's multimodal comprehension capabilities for significant advancement in web agents
- Novel evaluation protocol proposed by the research team for assessing web agents on open-ended tasks
- WebVoyager achieving an impressive 55.7% task success rate on real-world tasks from 15 popular websites, surpassing other setups
- Proposed automatic evaluation method showing an 85.3% agreement with human judgment, laying a foundation for future advancements in web agents

## Summary

In the realm of artificial intelligence, the evolution of large language models (LLMs) has ushered in a new era characterized by the emergence of autonomous applications in real-world scenarios. This development has spurred innovation in the creation of advanced web-based agents that have the potential to revolutionize how we interact with digital platforms. However, traditional web agents are often limited in their capabilities, as they typically only support one input modality and are evaluated in controlled environments such as simplified web simulators or static web snapshots. To address these limitations and bridge the gap between existing technology and real-world application, a team of researchers led by Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu have introduced WebVoyager. This groundbreaking Large Multimodal Model (LMM) powered web agent is designed to complete user instructions end-to-end by interacting with actual websites in a multimodal fashion. By leveraging the robust multimodal comprehension capabilities of GPT-4V, WebVoyager represents a significant advancement in the field of web agents. Furthermore, the research team proposed a novel evaluation protocol for assessing the performance of web agents on open-ended tasks. To test WebVoyager's efficacy, they curated real-world tasks from 15 popular websites and found that the agent achieved an impressive 55.7% task success rate. This performance surpassed both GPT-4 (All Tools) and WebVoyager (text-only) setups, highlighting the exceptional practical applicability of WebVoyager. Moreover, through their proposed automatic evaluation method - which demonstrated an 85.3% agreement with human judgment - the researchers have laid a solid foundation for further advancements in developing web agents for real-world settings. The findings from this study not only showcase the capabilities of WebVoyager but also pave the way for future innovations in autonomous applications and intelligent agents on the web.