# LoRA+: Efficient Low Rank Adaptation of Large Models

**Authors:** Soufiane Hayou, Nikhil Ghosh, Bin Yu  
**arXiv:** [2402.12354](https://arxiv.org/abs/2402.12354)  
**Date:** 2024-02-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.12354v1/

## Key points

- The original LoRA approach leads to suboptimal performance in networks with large widths due to inefficient feature learning.
- Identical learning rates for adapter matrices A and B hinder effective feature extraction, as established through scaling arguments.
- The authors propose a novel algorithm called LoRA$+$ which assigns different learning rates to adapter matrices A and B in a carefully chosen ratio, enhancing performance by 1-2% and accelerating finetuning speed by up to approximately 2 times compared to standard LoRA.
- Standard LoRA setup falls short in the infinite-width limit scenario, while LoRA$+$ proves instrumental in improving feature learning efficiency under low rank adaptation in this limit.
- The methodology presented is model-agnostic and applicable to general neural network architectures, emphasizing the significance of tailored learning rate adjustments through LoRA$+$.

## Summary

, , , , 

In this paper, the authors delve deeper into the concept of Low Rank Adaptation (LoRA) and its implications for finetuning large models with high embedding dimensions. They demonstrate that the original LoRA approach leads to suboptimal performance due to inefficient feature learning in networks with large widths. By employing scaling arguments, they establish that using identical learning rates for adapter matrices A and B hinders effective feature extraction. To address this limitation, the authors propose a novel algorithm called LoRA$+$, which rectifies the suboptimality of LoRA by assigning different learning rates to adapter matrices A and B in a carefully chosen ratio. Through extensive experiments, they showcase that LoRA$+$ enhances performance by 1-2% and accelerates finetuning speed by up to approximately 2 times compared to standard LoRA while maintaining the same computational cost. Furthermore, the study delves into the infinite-width limit of LoRA finetuning dynamics and reveals that standard LoRA setup falls short in this scenario. The introduction of LoRA$+$ proves instrumental in improving feature learning efficiency under low rank adaptation in this limit. The theoretical advancements are substantiated by empirical results spanning diverse language models and tasks. The methodology presented is model-agnostic and applicable to general neural network architectures. , , , , 

Considering a neural network structure comprising input embeddings Win, hidden weights Wl, output embeddings Wout, and mappings Fl defining layers across network depth L, pretraining on a dataset D is performed to accomplish specific tasks like next token prediction. The authors emphasize that pretraining involves minimizing an empirical loss function over training data pairs (x,y), where x represents input features from Rd-dimensional space and y denotes corresponding labels. Overall, this comprehensive analysis sheds light on the intricacies of low rank adaptation strategies in large-scale models and underscores the significance of tailored learning rate adjustments through LoRA$+$ for optimizing feature extraction efficiency during model finetuning processes.