# DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning

**Authors:** Abhay Zala, Han Lin, Jaemin Cho, Mohit Bansal  
**arXiv:** [2310.12128](https://arxiv.org/abs/2310.12128)  
**Date:** 2023-10-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.12128v2/

## Key points

- Text-to-image (T2I) models have advanced, but there is a gap in generating diagrams
- Diagrams are complex visual representations with interconnected objects, text labels, and arrows/lines
- Current T2I models struggle with fine-grained object layout and clear text labels for diagrams
- DiagrammerGPT is a two-stage framework using Large Language Models (LLMs) for diagram generation
- In the first stage, LLMs generate and refine "diagram plans" through an iterative process
- The second stage involves a diagram generator and text label rendering module to create coherent diagrams based on plans
- Evaluation on AI2D-Caption dataset shows DiagrammerGPT outperforms existing T2I models
- Authors Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal contributed significantly to the research effort

## Summary

Text-to-image (T2I) generation has made significant progress in recent years, but there is still a notable gap in the field when it comes to generating diagrams using T2I models. Diagrams are important visual representations that convey information through complex structures and spatial arrangements, including interconnected objects, text labels, and directional arrows/lines. However, current state-of-the-art T2I models struggle with diagram generation due to limitations in controlling fine-grained object layout and rendering clear text labels for densely connected objects with intricate relations like arrows/lines. To address these challenges, an innovative two-stage text-to-diagram generation framework called DiagrammerGPT has been introduced. This framework utilizes Large Language Models (LLMs) to guide the layout and improve the accuracy of diagram generation. In the first stage, LLMs generate and refine "diagram plans" through an iterative process within a planner-auditor feedback loop. Then, in the second stage, a diagram generator named DiagramGLIGEN along with a text label rendering module is used to create diagrams with coherent text labels based on the established diagram plans. To evaluate this approach's performance, a meticulously annotated diagram dataset called AI2D-Caption has been developed on top of the AI2D dataset. The results show that DiagrammerGPT outperforms existing T2I models by producing more precise diagrams. Additionally, comprehensive analyses have been conducted on open-domain diagram generation, multi-platform vector graphic diagram creation, human-in-the-loop editing processes, and multimodal planner/auditor LLM approaches. The authors Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal have made significant contributions to this research effort titled "DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning," showcasing their dedication to advancing text-to-diagram generation methodologies for enhanced visual representation of information.