# MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens

**Authors:** Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Kumar Guha, Matt Jordan, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, Ran Xu, Yejin Choi, Ludwig Schmidt  
**arXiv:** [2406.11271](https://arxiv.org/abs/2406.11271)  
**Date:** 2024-06-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.11271v1/

## Key points

- Multimodal interleaved datasets are crucial for training cutting-edge large multimodal models (LMMs)
- MINT-1T is the most extensive and diverse open-source Multimodal INTerleaved dataset to date
- MINT-1T contains one trillion text tokens and three billion images, representing a 10x scale-up from existing datasets
- The dataset includes previously untapped sources like PDFs and ArXiv papers
- Handling larger document sizes and preserving the original ordering of images and text poses significant engineering challenges in scaling multimodal interleaved datasets
- Data curation process for MINT-1T has been shared with the community to facilitate further research and development
- LMMs trained on MINT-1T show performance comparable to models trained on OBELICS
- Documents sourced from PDFs and ArXiv tend to be longer than HTML documents in MINT-1T compared to OBELICS
- Majority of documents in OBELICS are related to Humanities and Social Sciences, while Science-related documents dominate in MINT-1T's HTML subset
- Model experiments using MINT-1T will provide valuable insights into large multimodal model capabilities
- Release of data and code for MINT-T on GitHub aims to promote collaboration and innovation within the research community focused on multimodal learning

## Summary

, , , , 

Multimodal interleaved datasets, featuring free-form sequences of images and text, are essential for training cutting-edge large multimodal models (LMMs). Despite the rapid advancement of open-source LMMs, there is a significant shortage of large-scale, diverse open-source multimodal interleaved datasets. In response to this need, MINT-1T has been introduced as the most extensive and diverse open-source Multimodal INTerleaved dataset to date. With one trillion text tokens and three billion images, MINT-1T represents a 10x scale-up from existing open-source datasets. Additionally, previously untapped sources such as PDFs and ArXiv papers have been included in this dataset. Scaling multimodal interleaved datasets presents a significant engineering challenge due to the handling of larger document sizes and the preservation of the original ordering of images and text. The data curation process for MINT-1T has been shared with the community to facilitate further research and development in this area. Experimental results demonstrate that LMMs trained on MINT-1T exhibit performance comparable to models trained on the leading dataset OBELICS. In analyzing the composition of documents within MINT-1T compared to OBELICS, it was found that documents sourced from PDFs and ArXiv tend to be longer than HTML documents. Furthermore, a comparison of document domain distribution revealed interesting trends, such as the majority of documents in OBELICS being related to Humanities and Social Sciences, which differs from MINT-1T's HTML subset where Science-related documents dominate. Moving forward, model experiments utilizing MINT-1T will provide valuable insights into the capabilities of large multimodal models trained on this extensive dataset. The release of data and code for MINT-1T on GitHub will further promote collaboration and innovation within the research community focused on multimodal learning.