# The Impact of Initialization on LoRA Finetuning Dynamics

**Authors:** Soufiane Hayou, Nikhil Ghosh, Bin Yu  
**arXiv:** [2406.08447](https://arxiv.org/abs/2406.08447)  
**Date:** 2024-06-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.08447v1/

## Key points

- Study on the impact of initialization in Low Rank Adaptation (LoRA)
- Significance of starting from a pretrained model for finetuning
- Two initialization schemes: setting B to zero and A to random values
- Initializing B to zero and A to random consistently outperforms alternative scheme
- Ability to accommodate larger learning rates without causing output instability
- Critical role of initialization strategies in optimizing model performance

## Summary

In their paper titled "The Impact of Initialization on LoRA Finetuning Dynamics," Soufiane Hayou, Nikhil Ghosh, and Bin Yu delve into the role of initialization in Low Rank Adaptation (LoRA). The study focuses on the significance of starting from a pretrained model for finetuning and explores two initialization schemes. These schemes involve setting either B or A to zero and random values respectively. Through theoretical analysis and extensive experiments on Large Language Models (LLMs), the authors demonstrate that initializing B to zero and A to random consistently outperforms the alternative scheme. This is due to its ability to accommodate larger learning rates without causing output instability. The research highlights the critical role of initialization strategies in optimizing model performance and provides valuable insights for practitioners seeking to enhance finetuning processes for LLMs.