# LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

**Authors:** Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, Yu Qiao  
**arXiv:** [2304.15010](https://arxiv.org/abs/2304.15010)  
**Date:** 2023-04-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.15010v1/

## Key points

- Efficiently transforming large language models (LLMs) into instruction followers has gained significant attention in recent years
- Training LLMs for multi-modal reasoning remains relatively unexplored
- LLaMA-Adapter struggled to generalize well to open-ended visual instructions and lagged behind GPT-4
- LLaMA-Adapter V2 is a parameter-efficient visual instruction model that addresses these limitations
- LLaMA-Adapter V2 unlocks more learnable parameters such as norm, bias, and scale to distribute the instruction-following ability across the entire LLaMA model besides adapters
- An early fusion strategy is proposed to feed visual tokens only into the early layers of the LLM, contributing to better incorporation of visual knowledge
- A joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters
- Additional expert models such as captioning/OCR systems are incorporated into LLaMA Adapter during inference to further enhance its image understanding capability without incurring training costs.
- Compared to the original LLaMA Adapter, LLaMA Adapter V2 can perform open ended multi modal instructions by introducing only 14M parameters over LLaMA.
- The newly designed framework also exhibits stronger language-only instruction following capabilities and excels in chat interactions.

## Summary

The field of efficiently transforming large language models (LLMs) into instruction followers has gained significant attention in recent years, but training LLMs for multi-modal reasoning remains relatively unexplored. While the recent LLaMA-Adapter showed promise in handling visual inputs with LLMs, it struggled to generalize well to open-ended visual instructions and lagged behind GPT-4. In this paper, the authors present LLaMA-Adapter V2, a parameter-efficient visual instruction model that addresses these limitations. The first improvement made in LLaMA-Adapter V2 is unlocking more learnable parameters such as norm, bias, and scale. These parameters distribute the instruction-following ability across the entire LLaMA model besides adapters. Secondly, an early fusion strategy is proposed to feed visual tokens only into the early layers of the LLM, contributing to better incorporation of visual knowledge. Thirdly, a joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters. This approach effectively alleviates interference between image-text alignment and instruction following tasks and achieves strong multi-modal reasoning with only a small scale image-text and instruction dataset. During inference, additional expert models such as captioning/OCR systems are incorporated into LLaMA Adapter to further enhance its image understanding capability without incurring training costs. Compared to the original LLaMA Adapter, LLaMA Adapter V2 can perform open ended multi modal instructions by introducing only 14M parameters over LLaMA. The newly designed framework also exhibits stronger language only instruction following capabilities and excels in chat interactions. Overall, this paper presents a promising solution for efficient transformation of large language models into effective instruction followers for multi modal reasoning tasks while addressing some key limitations of previous approaches. The code and models are available at https://github.com/ZrrSkywalker/LLaMA Adapter for further exploration and implementation.