# Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

**Authors:** Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, Zhe Gan  
**arXiv:** [2410.18967](https://arxiv.org/abs/2410.18967)  
**Date:** 2024-10-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.18967v1/

## Key points

- Ferret-UI 2 is a multimodal large language model (MLLM) designed for universal UI understanding across platforms
- Support for multiple platform types for seamless adaptation
- Enhanced high-resolution perception through adaptive scaling techniques
- Advanced task training data generation using GPT-4o and visual prompting techniques
- Empirical experiments show superior performance of Ferret-UI 2 compared to its predecessor on various tasks and platforms
- Demonstrates robust cross-platform transfer capabilities, setting a new standard in universal UI understanding

## Summary

Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

In the realm of user interface (UI) understanding, building a generalist model that can effectively navigate the complexities of platform diversity, resolution variation, and data limitations is no small feat. However, Zhangheng Li and his team have introduced Ferret-UI 2 as a groundbreaking solution to this challenge in their recent paper. Ferret-UI 2 stands out as a multimodal large language model (MLLM) specifically crafted to excel in universal UI understanding across a diverse array of platforms including iPhone, Android, iPad, Webpage, and AppleTV. It builds upon its predecessor Ferret-UI with three key innovations that elevate its capabilities to new heights. Firstly, Ferret-UI 2 boasts support for multiple platform types which enables it to seamlessly adapt to various interfaces with ease. Secondly, the model enhances high-resolution perception through adaptive scaling techniques ensuring optimal performance regardless of screen size or display quality. Lastly, Ferret-UI 2 leverages advanced task training data generation powered by GPT-4o with set-of-mark visual prompting techniques to facilitate complex user-centered interactions. The empirical experiments conducted on referring tasks, grounding exercises, user-centric advanced tasks spanning nine subtasks across five platforms as well as evaluations on the GUIDE next-action prediction dataset and GUI-World multi-platform benchmark showcase Ferret-UI 2's superior performance compared to its predecessor. Not only does Ferret-UI 2 outperform Ferret-UI significantly but it also demonstrates robust cross-platform transfer capabilities. In conclusion, With its ability to perform intricate user interactions across various platforms with precision and efficiency, Ferret-UI 2 sets a new standard in universal UI understanding.