# LoRA: Low-Rank Adaptation of Large Language Models

**Authors:** Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen  
**arXiv:** [2106.09685](https://arxiv.org/abs/2106.09685)  
**Date:** 2021-06-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.09685v2/

## Key points

- Introduction of Low-Rank Adaptation (LoRA) as a novel approach for adapting large language models
- LoRA involves freezing pre-trained model weights and introducing trainable rank decomposition matrices into each layer of the Transformer architecture
- Significant reduction in trainable parameters for downstream tasks compared to traditional fine-tuning methods, with up to 10,000 times fewer parameters and three times less GPU memory requirements
- Comparable or superior model quality performance on popular language models like RoBERTa, DeBERTa, GPT-2, and GPT-3
- Higher training throughput and no additional inference latency compared to adapter-based approaches
- Empirical investigation into rank-deficiency in language model adaptation to support the effectiveness of LoRA
- Availability of a package with implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 on GitHub

## Summary

The paper "LoRA: Low-Rank Adaptation of Large Language Models" by authors Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen introduces a novel approach to adapting large language models for specific tasks or domains. The traditional method of full fine-tuning on pre-trained models like GPT-3 175B becomes impractical due to the high computational cost associated with deploying multiple instances of fine-tuned models. In response to this challenge, the authors propose Low-Rank Adaptation (LoRA), which involves freezing the pre-trained model weights and introducing trainable rank decomposition matrices into each layer of the Transformer architecture. By implementing LoRA, the number of trainable parameters for downstream tasks can be significantly reduced compared to traditional fine-tuning methods. For instance, when compared to fine-tuning GPT-3 175B with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and decrease GPU memory requirements by three times. Despite having fewer trainable parameters, LoRA demonstrates comparable or superior model quality performance on popular language models such as RoBERTa, DeBERTa, GPT-2 and GPT-3. Furthermore, LoRA offers higher training throughput and does not introduce additional inference latency like adapter-based approaches. The paper also includes an empirical investigation into rank-deficiency in language model adaptation to provide insights into the effectiveness of LoRA. To facilitate the integration of LoRA with PyTorch models,the authors have released a package along with implementations and model checkpoints for RoBERTa ,DeBERTa,and GPT-2 on GitHub. In summary,"LoRA: Low-Rank Adaptation of Large Language Models" presents a promising solution for efficiently adapting large language models to specific tasks or domains while maintaining high performance levels and reducing computational costs.