# LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

**Authors:** Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Yu Qiao  
**arXiv:** [2303.16199](https://arxiv.org/abs/2303.16199)  
**Date:** 2023-03-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.16199v2/

## Key points

- LLaMA-Adapter is a lightweight adaptation method for fine-tuning the LLaMA language model into an instruction-following model.
- It leverages 52K self-instruct demonstrations and introduces only 1.2M learnable parameters on top of the frozen LLaMA 7B model.
- The fine-tuning process can be completed in less than an hour using 8 A100 GPUs.
- Learnable adaption prompts are incorporated at higher transformer layers to provide instructional cues.
- A zero-initialized attention mechanism with zero gating is used to inject new instructional cues while preserving pre-trained knowledge.
- LLaMA-Adapter generates high-quality responses comparable to Alpaca with fully fine-tuned 7B parameters.
- It can be extended to multi-modal instructions for learning image-conditioned LLaMA models, improving reasoning performance on benchmarks like ScienceQA and COCO Caption.
- The zero-initialized attention mechanism used in LLaMA Adapter is evaluated for fine-tuning other pre-trained models like ViT and RoBERTa on traditional vision and language tasks.
- The code for LLaMA Adapter is publicly available at https://github.com/OpenGVLab/LLaMA-Adapter.

## Summary

LLaMA-Adapter is a lightweight adaptation method that efficiently fine-tunes the LLaMA language model into an instruction-following model. By leveraging 52K self-instruct demonstrations, LLaMA-Adapter introduces only 1.2M learnable parameters on top of the frozen LLaMA 7B model and can be fine-tuned in less than an hour using 8 A100 GPUs. The approach involves incorporating a set of learnable adaption prompts, which are prepended to the word tokens at higher transformer layers. Additionally, a zero-initialized attention mechanism with zero gating is proposed to adaptively inject new instructional cues into LLaMA while preserving its pre-trained knowledge. With this efficient training process, LLaMA-Adapter demonstrates the ability to generate high quality responses comparable to Alpaca with fully fine tuned 7B parameters. Moreover, the approach can be extended to multi modal instructions for learning image conditioned LLaMA models, leading to superior reasoning performance on benchmarks such as ScienceQA and COCO Caption. Furthermore, the zero initialized attention mechanism used in LLaMA Adapter is evaluated for fine tuning other pre trained models like ViT and RoBERTa on traditional vision and language tasks. This evaluation showcases the superior generalization capacity of the approach. The code for LLaMA Adapter is publicly available at https://github.com/OpenGVLab/LLaMA-Adapter.