# DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation

**Authors:** Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, Ali Ghodsi  
**arXiv:** [2210.07558](https://arxiv.org/abs/2210.07558)  
**Date:** 2022-10-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.07558v2/

## Key points

- Pretrained models (PMs) have grown in size, making fine-tuning them resource-intensive
- Low-rank adapters (LoRA) freeze main pretrained weights and use truncated singular value decomposition (SVD) modules called LoRA blocks that are learnable
- LoRA blocks suffer from fixed size and optimizing their rank requires exhaustive search
- Dynamic low-rank adaptation (DyLoRA) trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training
- DyLoRA can train dynamic search-free models at least 4 to 7 times faster than LoRA without compromising performance significantly
- DyLoRa's models perform consistently well over a much larger range of ranks compared to LoRA
- This approach offers an efficient way to fine-tune PMs while reducing computational costs and time requirements for optimization tasks.

## Summary

In recent years, the size of pretrained models (PMs) has grown significantly, making fine-tuning them a resource-intensive task. To address this issue, low-rank adapters (LoRA) have been proposed, which freeze the main pretrained weights of the model and introduce truncated singular value decomposition (SVD) modules called LoRA blocks that are learnable. However, LoRA blocks suffer from two major problems: their size is fixed and cannot be modified after training, and optimizing their rank requires an exhaustive search. In response to these challenges, a new technique called dynamic low-rank adaptation (DyLoRA) has been introduced in this work. DyLoRA trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. The authors evaluated their solution on various natural language understanding tasks such as GLUE benchmark and language generation tasks like E2E, DART, and WebNLG using different pretrained models such as RoBERTa and GPT with varying sizes. The results show that DyLoRA can train dynamic search-free models at least 4 to 7 times faster than LoRA without compromising performance significantly. Furthermore, DyLoRa's models perform consistently well over a much larger range of ranks compared to LoRA. This approach offers an efficient way to fine-tune PMs while reducing computational costs and time requirements for optimization tasks.