# Flora: Low-Rank Adapters Are Secretly Gradient Compressors

**Authors:** Yongchang Hao, Yanshuai Cao, Lili Mou  
**arXiv:** [2402.03293](https://arxiv.org/abs/2402.03293)  
**Date:** 2024-02-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.03293v1/

## Key points

- LoRA method aims to reduce memory usage in large neural networks by training fewer parameters and decreasing optimization states.
- Flora is introduced as a novel approach to address limitations of LoRA, leveraging random projection to achieve high-rank updates while maintaining model performance.
- Flora allows for sublinear space complexity in storing optimization states.
- Experiments involve fine-tuning pre-trained models using gradient accumulation and training from scratch with momentum techniques.
- Effectiveness is evaluated using ROUGE scores for summarization tasks and SacreBLEU scores for translation tasks.
- Peak memory usage is monitored, and comparisons are made with competing approaches such as Adafactor.
- Experiments are conducted across different model architectures (T5 and GPT-2 series) on tasks like summarization and translation.
- Efficiency of Flora in optimizing memory usage without compromising model performance is demonstrated through testing various rank values for small and large models, showing significant improvements compared to existing methods.

## Summary

In this study, we delve into the dynamics of the low-rank adaptation (LoRA) method and introduce Flora as a novel approach to address its limitations. LoRA aims to reduce memory usage in large neural networks by training fewer parameters and has shown promise in decreasing optimization states. However, it comes with the drawback of limiting model performance due to its restriction on weight update matrices. To overcome this issue, Flora leverages random projection to approximate LoRA and achieve high-rank updates by resampling projection matrices. This allows for maintaining model performance while enjoying sublinear space complexity in storing optimization states. Our experiments involve fine-tuning a pre-trained model using gradient accumulation and training from scratch with momentum techniques. We evaluate the effectiveness of our approach using ROUGE scores for summarization tasks and SacreBLEU scores for translation tasks. Additionally, we monitor peak memory usage and compare our method with competing approaches such as Adafactor. We conduct experiments across different model architectures, including T5 and GPT-2 series models, on tasks like summarization and translation. By testing various rank values for small and large models, we demonstrate the efficiency of Flora in optimizing memory usage without compromising model performance. Our results show significant improvements in both memory savings and task performance compared to existing methods.