Lexi: Self-Supervised Learning of the UI Language

In the field of user interface (UI) language understanding, there is a need for generic visio-linguistic representations of UI screens and their components that can be used in various applications such as accessibility, voice navigation, and task automation. While existing UI representation models rely on UI metadata which may not always be available or accessible, a new approach has been proposed to leverage data from instruction manuals and how-to guides to learn these representations. This approach involves the use of a pre-trained vision and language model called Lexi, which is designed to handle the unique features of UI screens such as their text richness and context sensitivity. To train Lexi, a new dataset called UICaption was curated consisting of 114k UI images paired with descriptions of their functionality. The performance of Lexi was evaluated on four tasks including UI action entailment, instruction-based UI image retrieval, grounding referring expressions, and UI entity recognition. The results showed that Lexi outperformed existing models in all tasks. Related work in this area includes multiple transformer-based architectures proposed to learn a single feature space from visual and language inputs. However, unlike pre-training language models using unlimited natural language texts like Wikipedia, visio-linguistic (VL) models use high-quality paired datasets such as Conceptual Captions, SBU Captions, and MS COCO. This new approach extends prior architectures with a vision encoder designed to leverage the text richness of UI images. While instruction manuals are useful resources for learning how to operate an application's user interface (UI), they may not always be available or complete for certain application categories like shopping or transportation. Additionally, curating pre-training data from the web presents challenges in ensuring zero overlap between pre-training and downstream evaluation tasks. Future work should focus on expanding this method to different languages beyond English instructions and evaluating its performance on longer instructions. Pre-training the model requires significant GPU resources making it difficult to execute on edge devices like mobile phones raising privacy concerns; however it would be highly desirable for scenarios motivating this work such as accessibility support and voice navigation in mobile apps to execute and update the models locally. Overall, this new approach offers a promising solution for learning generic visio-linguistic representations of UI screens without relying on metadata while also avoiding overlap between pre-training data sources and downstream evaluation tasks.

Understanding User Interface Language with Lexi: A New Approach to Visio-Linguistic Representations

The user interface (UI) of an application is a critical component for its usability and accessibility. To enable voice navigation, task automation, and other applications, there is a need for generic visio-linguistic representations of UI screens and their components. While existing models rely on UI metadata which may not always be available or accessible, a new approach has been proposed to leverage data from instruction manuals and how-to guides to learn these representations.

Background

In recent years, multiple transformer-based architectures have been proposed to learn a single feature space from visual and language inputs. These architectures are typically pre-trained using large natural language datasets like Wikipedia; however this method does not work well with high quality paired datasets such as Conceptual Captions, SBU Captions, or MS COCO due to their limited size. To address this issue, Chen et al. propose a new approach that extends prior architectures with a vision encoder designed specifically for UI images in order to leverage the text richness of these images.

Lexi Model

To train Lexi, the authors curated UICaption - a dataset consisting of 114k UI images paired with descriptions of their functionality - which was used in conjunction with existing datasets such as Conceptual Captions (CC), SBU Caption (SBU), MS COCO (MSCOCO), Visual Genome (VG). The performance of Lexi was evaluated on four tasks including UI action entailment, instruction based image retrieval task (IR), grounding referring expressions (RE) task ,and entity recognition task(ER). The results showed that Lexi outperformed existing models in all tasks demonstrating its effectiveness at learning visio-linguistic representations without relying on metadata while also avoiding overlap between pre-training data sources and downstream evaluation tasks.

Challenges & Future Work

While instruction manuals are useful resources for learning how to operate an application's user interface (UI), they may not always be available or complete for certain application categories like shopping or transportation. Additionally curating pre-training data from web presents challenges in ensuring zero overlap between pre training and downstream evaluation tasks . Pre training the model requires significant GPU resources making it difficult execute on edge devices like mobile phones raising privacy concerns; however it would be highly desirable scenarios motivating this work such as accessibility support voice navigation in mobile apps execute update models locally . Future work should focus expanding method different languages beyond English instructions evaluating performance longer instructions .

Conclusion

This new approach offers promising solution learning generic visio linguistic representation UIs without relying metadata while also avoiding overlap between pre training data sources downstream evaluation tasks . With further development ,this could lead improved usability accessibility applications through better understanding user interfaces across various languages contexts .