# $VILA^2$: VILA Augmented VILA

**Authors:** Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jang Hyun Cho, Marco Pavone, Song Han, Hongxu Yin  
**arXiv:** [2407.17453](https://arxiv.org/abs/2407.17453)  
**Date:** 2024-07-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.17453v1/

## Key points

- Advancements in visual language models (VLMs) driven by large language models (LLMs)
- Introduction of self-augment step and specialist-augment step to enhance data quality and improve model performance
- Self-augmentation process involving recaptions of pretraining data and retraining from scratch
- Fine-tuning specialist VLMs with domain-specific expertise from self-augmented VLM for task-oriented recaptioning and retraining
- Creation of $VILA^2$ family of VLMs outperforming prior art across tasks and achieving state-of-the-art results on MMMU leaderboard
- Utilization of diverse set of images for OCR strength enrichment within the VILA framework, emphasizing text recognition, comprehension, and reasoning tasks
- Significant improvements in accuracy and performance metrics through self-augmentation techniques and specialist augmentation strategies

## Summary

, , , , 


In the rapidly evolving landscape of visual language models (VLMs), advancements have been primarily driven by the success of large language models (LLMs). While model architectures and training infrastructures continue to progress at a rapid pace, the aspect of data curation has remained relatively under-explored. To address this challenge, a novel approach is introduced in this work that incorporates a self-augment step and a specialist-augment step to iteratively enhance data quality and improve model performance. In the self-augment step, a VLM recaptions its own pretraining data to refine data quality, followed by retraining from scratch using this enhanced dataset to boost model performance. This iterative process can be repeated for multiple rounds until self-augmentation reaches saturation. Subsequently, several specialist VLMs with domain-specific expertise are fine-tuned from the self-augmented VLM to infuse specialist knowledge into the generalist VLM through task-oriented recaptioning and retraining. The culmination of these efforts results in the creation of $VILA^2$ (VILA-augmented-VILA), a family of VLMs that consistently outperforms prior art across a wide range of tasks and achieves new state-of-the-art results on the MMMU leaderboard among open-sourced models. Additionally, focusing on OCR strength enrichment, a diverse set of images containing textual content such as tables, charts, and documents are utilized to develop an OCR specialist within the VILA framework. Each image is annotated with QA pairs emphasizing text recognition, comprehension, and reasoning tasks to enhance OCR capabilities. Overall, through a combination of self-augmentation techniques and specialist augmentation strategies tailored towards specific domains like OCR processing, $VILA^2$ demonstrates significant improvements in accuracy and performance metrics compared to existing approaches. This comprehensive methodology not only showcases advancements in visual language modeling but also underscores the importance of meticulous data curation for pushing the boundaries of model capabilities in real-world applications.