# LLaVA-KD: A Framework of Distilling Multimodal Large Language Models

**Authors:** Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, Xiang Bai  
**arXiv:** [2410.16236](https://arxiv.org/abs/2410.16236)  
**Date:** 2024-10-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.16236v1/

## Key points

- Large Language Models (LLM) have led to the development of Multimodal Large Language Models (MLLM) that combine visual and linguistic understanding.
- Challenges arise in deploying large MLLMs due to their size and computational complexity, leading to the development of Small-scale MLLMs (s-MLLM) for more efficient processing.
- The LLaVA-KD framework facilitates knowledge transfer from large-scale MLLMs to small-scale ones through Multimodal Distillation (MDist) and Relation Distillation (RDist).
- A three-stage training scheme within the LLaVA-KD framework includes Distilled Pre-Training, Supervised Fine-Tuning, and Distilled Fine-Tuning to optimize s-MLLM performance.
- Results show significant performance improvements using the LLaVA-KD approach without requiring changes to the architecture of small-scale models.
- The research paper "LLaVA-KD: A Framework of Distilling Multimodal Large Language Models" by Yuxuan Cai et al. is currently under review and aims to advance multimodal understanding in AI systems.

## Summary

In the realm of artificial intelligence and machine learning, the emergence of Large Language Models (LLM) has sparked a wave of innovation and exploration into Multimodal Large Language Models (MLLM). These MLLMs aim to bridge the gap between visual and linguistic understanding, offering a unified approach to processing information. However, the sheer size and computational complexity of these models present challenges when it comes to deploying them in resource-constrained environments. To address this issue, researchers have turned their attention towards developing Small-scale MLLMs (s-MLLM) that can retain the capabilities of their larger counterparts while being more computationally efficient. Despite the potential benefits of s-MLLMs, there is often a trade-off in performance compared to their larger counterparts. In light of these challenges, a novel framework known as LLaVA-KD has been proposed to facilitate knowledge transfer from large-scale MLLMs to small-scale ones. This framework introduces two key components: Multimodal Distillation (MDist) aims to minimize the discrepancies between visual-textual output distributions of large and small models, while Relation Distillation (RDist) focuses on transferring the ability of large models to model correlations between visual features. Moreover, a three-stage training scheme is outlined within the LLaVA-KD framework to fully leverage the potential of s-MLLMs. This includes Distilled Pre-Training for aligning visual-textual representations, Supervised Fine-Tuning for enhancing multimodal understanding, and Distilled Fine-Tuning for further transferring capabilities from large models. The results obtained from applying this approach demonstrate significant improvements in performance without necessitating alterations to the architecture of small-scale models. Extensive experiments and ablation studies have validated the effectiveness of each proposed component within the LLaVA-KD framework. The authors behind this groundbreaking work include Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, and Xiang Bai. Their research paper titled "LLaVA-KD: A Framework of Distilling Multimodal Large Language Models" is currently under review and promises to make valuable contributions to advancing multimodal understanding in AI systems.