# Kosmos-2.5: A Multimodal Literate Model

**Authors:** Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei  
**arXiv:** [2309.11419](https://arxiv.org/abs/2309.11419)  
**Date:** 2023-09-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.11419v1/

## Key points

- Kosmos-2.5 is a multimodal literate model for machine reading of text-intensive images.
- It has been pre-trained on approximately 27.6 million pages of document images from various sources.
- The model excels in generating spatially-aware text blocks and producing structured text output in markdown format.
- It uses a shared Transformer architecture, task-specific prompts, and flexible text representations.
- Kosmos-2.5 has been evaluated on document-level text recognition and image-to-markdown text generation tasks with promising results.
- The model can be easily adapted for other text-intensive image understanding tasks through supervised fine-tuning.
- The training data includes diverse sources such as arXiv papers, PowerPoint slides, general PDFs, and web screenshots.
- For structured text output in markdown format, the model leverages data from README files, DOCX pages converted to markdown format, LATEX code converted to markdown information, and HTML files converted to markdown format.
- Different processing workflows are used depending on the type of data involved: scanned document images are processed using the Microsoft Read API, ArXiv papers, PowerPoint slides, and general PDFs are compiled into PDF files and parsed using the PyMuPDF parser, webpage screenshots are collected using Playwright and extracting HTML content using the lxml library, README files are collected from GitHub projects and converted to HTML using Pandoc with images obtained from the generated HTML content using wkhtmltopdf.

## Summary

We present Kosmos-2.5, a multimodal literate model for machine reading of text-intensive images. Pre-trained on approximately 27.6 million pages of document images, including data from arXiv papers, PowerPoint slides, general PDFs and web screenshots, Kosmos-2.5 excels in generating spatially-aware text blocks and producing structured text output in markdown format. The model achieves this through a shared Transformer architecture, task-specific prompts and flexible text representations. It has been evaluated on end-to-end document-level text recognition and image-to-markdown text generation tasks with promising results. In addition to its capabilities, Kosmos-2.5 can be easily adapted for other text-intensive image understanding tasks through supervised fine tuning making it a versatile tool for real world applications involving text rich images. The model's training data includes diverse sources such as arXiv papers (20.9 million pages), PowerPoint slides (6.2 million pages), general PDFs (155.2 million pages) and web screenshots (almost 100 million pages). For structured text output in markdown format the model leverages data from README files (2.9 million files), DOCX pages converted to markdown format (1.1 million pages), LATEX code converted to markdown information (3.7 million pages) and HTML files converted to markdown format (6.3 million files). The pre training data goes through different processing workflows depending on the type of data involved; scanned document images are processed using the Microsoft Read API to extract text and layout information while ArXiv papers, PowerPoint slides and general PDFs are compiled into PDF files and parsed using the PyMuPDF parser to efficiently extract text and layout information; webpage screenshots are collected by accessing specified URLs using Playwright and extracting HTML content using the lxml library to obtain a Document Object Model (DOM) tree representation of the webpage; README files are collected from GitHub projects and converted to HTML using Pandoc with images obtained from the generated HTML content using wkhtmltopdf .