# mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

**Authors:** Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, Fei Huang  
**arXiv:** [2304.14178](https://arxiv.org/abs/2304.14178)  
**Date:** 2023-04-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.14178v1/

## Key points

- The study introduces mPLUG-Owl, a training paradigm that enhances the multi-modal abilities of large language models (LLMs) for multi-modal generation.
- The approach involves modularized learning of foundation LLM, a visual knowledge module and a visual abstractor module to support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration.
- The training paradigm employs a two-stage method for aligning image and text which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM.
- Experimental results show that mPLUG-Owl outperforms existing multi-modal models in instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability.
- Unexpected abilities such as multi-image correlation and scene text understanding were observed making it possible to leverage them for harder real scenarios such as vision only document comprehension.
- Furthermore, mPLUG-Owl performs well in open ended creation tasks such as poetry lyrics advertisements based on images but requires further exploration for more functional practical creations.
- Code snippets used in this study are available at https://github.com/X-PLUG/mPLUG-Owl along with pre-trained models for evaluation purposes.

## Summary

This study introduces mPLUG-Owl, a novel training paradigm that enhances the multi-modal abilities of large language models (LLMs) for multi-modal generation. The approach involves modularized learning of foundation LLM, a visual knowledge module and a visual abstractor module to support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm employs a two-stage method for aligning image and text which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM. In the first stage, the visual knowledge module and abstractor module are trained with a frozen LLM module to align the image and text. In the second stage, language-only and multi-modal supervised datasets are used to jointly fine-tune a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing the visual knowledge module. The study also presents an evaluation set called OwlEval that tests visually related instructions. Experimental results show that mPLUG-Owl outperforms existing multi-modal models in instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability. Additionally, unexpected abilities such as multi-image correlation and scene text understanding were observed making it possible to leverage them for harder real scenarios such as vision only document comprehension. Furthermore, mPLUG-Owl performs well in open ended creation tasks such as poetry lyrics advertisements based on images but requires further exploration for more functional practical creations. Overall this study proposes an innovative approach to enhance LLMs' multi modal abilities through modularized learning that can facilitate diverse unimodal and multimodal abilities through modality collaboration. The code snippets used in this study are available at https://github.com/X-PLUG/mPLUG-Owl along with pre trained models for evaluation purposes.