# AppAgent v2: Advanced Agent for Flexible Mobile Interactions

**Authors:** Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, Yunchao Wei  
**arXiv:** [2408.11824](https://arxiv.org/abs/2408.11824)  
**Date:** 2024-08-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.11824v3/

## Key points

- Introduction of a multimodal agent framework combining parser with visual features for enhanced interaction with GUIs and adaptability to new tasks
- Development of structured storage format and RAG technology for real-time updates and access to knowledge base, improving adaptability and decision-making precision
- Extensive empirical testing demonstrating effectiveness across various smartphone applications, validating adaptability, user-friendliness, and efficiency in real-world scenarios

## Summary

With the advancement of Multimodal Large Language Models (MLLM), LLM-driven visual agents are revolutionizing software interfaces, particularly those with graphical user interfaces. This study presents a cutting-edge LLM-based multimodal agent framework tailored for mobile devices. The framework, designed to navigate mobile devices, mimics human-like interactions and constructs a versatile action space that enhances adaptability across various applications such as parser, text, and vision descriptions. The agent operates through two main phases: exploration and deployment. During the exploration phase, the functionalities of user interface elements are documented either through agent-driven or manual explorations into a customized structured knowledge base. In the deployment phase, Rapid Automatic Generalization (RAG) technology facilitates efficient retrieval and updates from this knowledge base, empowering the agent to execute tasks effectively and accurately. This includes performing complex multi-step operations across diverse applications, showcasing the framework's adaptability and precision in handling customized task workflows. The effectiveness of our agent was validated through tests on three distinct benchmarks covering tasks across numerous applications. Quantitative results and user studies confirm the superiority and robustness of our approach. In summary, this paper makes significant contributions
- Introduction of a multimodal agent framework combining parser with visual features to create a flexible action space for enhanced interaction with GUIs and improved adaptability to new environmental tasks. - Development of a novel structured storage format coupled with RAG technology for adaptive real-time updates and access to the knowledge base, enhancing the agent's adaptability and decision-making precision. - Extensive empirical testing demonstrating the agent's effectiveness across various smartphone applications, validating its adaptability, user-friendliness, and efficiency in real-world scenarios. The detailed description of our multimodal agent framework is provided in Section 2 where we outline two primary phases: exploration and deployment. The agent analyzes the current GUI along with task requirements at each round to generate observations, thoughts, actions, and summaries. The summary acts as memory carried over to ensure continuity throughout task execution. Implemented on Android environment using Android Studio emulator, our multimodal agent interacts with mobile phones by invoking commands through AndroidController based on analyzing structured data parsing information from GUI interface combined with OCR and detection models for extracting detailed information from screenshots. This setup enables efficient performance within dynamic mobile environments integrating advanced recognition capabilities with intelligent decision-making processes based on interpreted data from UI. Commands transformed into instructions are executed by Android system via AndroidController ensuring precise command execution for efficient task performance within Android environment. Agent interactions during both exploration and execution phases involve translating human commands or LLM outputs into instructions recognized by Android system including TapButton for tap actions on UI elements specified by number identifier or visual features; Text for simulating typing; LongPress for prolonged press; Swipe for executing swipe actions in specified directions; Back simulating device's back button return to previous UI state. In conclusion, our refined detailed longer summary highlights the innovative multimodal agent framework's efficacy in enhancing interaction with GUIs on mobile devices through advanced technologies like MLLM-driven visual agents combined with RAG technology for adaptive knowledge base updates facilitating superior performance across various smartphone applications in real-world scenarios.