# Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

**Authors:** Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, Yu Su  
**arXiv:** [2410.05243](https://arxiv.org/abs/2410.05243)  
**Date:** 2024-10-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.05243v1/

## Key points

- Multimodal large language models (MLLMs) are revolutionizing GUI agents by enabling them to transition from controlled simulations to real-world applications.
- The robustness of GUI agents heavily relies on their grounding capability, which is currently predominantly text-based but introduces noise and computational overhead.
- Authors propose an approach where agents perceive their environment visually and operate at the pixel level on the GUI for a more human-like embodiment.
- Visual grounding models accurately map diverse referring expressions of GUI elements to their coordinates on the interface across different platforms.
- UGround, a universal visual grounding model tailored for GUI agents, significantly outperforms existing models by up to 20% in empirical results across various benchmarks.
- Agents equipped with UGround surpass state-of-the-art counterparts despite relying solely on visual perception, showcasing the potential for more intuitive human-computer interactions.

## Summary

, , , , 

In their paper titled "Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents," authors Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su delve into the transformative impact of multimodal large language models (MLLMs) on graphical user interface (GUI) agents. These MLLMs are revolutionizing the capabilities of GUI agents by enabling them to transition from controlled simulations to complex real-world applications across various platforms. The effectiveness of these agents heavily relies on the robustness of their grounding capability. The current landscape of GUI agents predominantly relies on text-based representations like HTML or accessibility trees. While these representations have proven useful, they often introduce noise, incompleteness, and increased computational overhead. To address these limitations and advocate for a more human-like embodiment for GUI agents, the authors propose an approach where agents perceive their environment entirely visually and directly operate at the pixel level on the GUI. Central to this approach are visual grounding models that accurately map diverse referring expressions of GUI elements to their coordinates on the interface across different platforms. The authors demonstrate the effectiveness of a simple recipe involving web-based synthetic data and slight adaptations to the LLaVA architecture in training such visual grounding models. They curate an extensive dataset for GUI visual grounding comprising 10 million GUI elements and their corresponding referring expressions across 1.3 million screenshots. Utilizing this dataset, they train UGround – a robust universal visual grounding model tailored for GUI agents. Empirical results across six benchmarks spanning three categories (grounding, offline agent performance, online agent performance) showcase that UGround significantly outperforms existing visual grounding models for GUI agents by up to 20% absolute improvement. Moreover, agents equipped with UGround surpass state-of-the-art counterparts despite relying solely on visual perception while existing models incorporate additional text-based input. These findings underscore the feasibility and potential of GUI agents that navigate digital interfaces akin to how humans interact with them. The research not only advances the field of graphical user interface technology but also paves the way for more intuitive and efficient human-computer interactions in diverse application domains.