# CogAgent: A Visual Language Model for GUI Agents

**Authors:** Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, Jie Tang  
**arXiv:** [2312.08914](https://arxiv.org/abs/2312.08914)  
**Date:** 2023-12-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.08914v2/

## Key points

- Authors developed CogAgent, an 18-billion-parameter visual language model (VLM) for GUI understanding and navigation
- CogAgent uses both low-resolution and high-resolution image encoders to accurately identify elements and text within GUIs at a resolution of 1120*1120
- Demonstrates state-of-the-art performance in various benchmarks related to text-rich and general visual question answering (VQA)
- Surpasses LLM-based methods like Mind2Web and AITW in PC and Android GUI navigation tasks using only screenshots as input
- Model and corresponding codes are accessible at https://github.com/THUDM/CogVLM

## Summary

, , , , 

In their paper titled "CogAgent: A Visual Language Model for GUI Agents," authors Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang address the increasing reliance on digital devices with graphical user interfaces (GUIs) like computer and smartphone screens. The authors introduce CogAgent - an 18-billion-parameter visual language model (VLM) designed specifically for GUI understanding and navigation to enhance automation levels. CogAgent leverages both low-resolution and high-resolution image encoders to accurately identify even the smallest page elements and text within GUIs at a resolution of 1120*1120. This versatile visual language model demonstrates state-of-the-art performance in various benchmarks related to text-rich and general visual question answering (VQA), including VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA MM-Vet,and POPE. CogAgent surpasses LLM-based methods like Mind2Web and AITW in PC and Android GUI navigation tasks solely using screenshots as input. The comprehensive study conducted by the authors culminates in making both the model itself and its corresponding codes accessible at https://github.com/THUDM/CogVLM. With its innovative approach to visual language modeling tailored for GUI applications,CogAgent represents a significant step forward in optimizing digital interactions on various devices through enhanced understanding of graphical user interfaces.