# PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

**Authors:** Nadav Benedek, Lior Wolf  
**arXiv:** [2401.11316](https://arxiv.org/abs/2401.11316)  
**Date:** 2024-01-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.11316v1/

## Key points

- PRILoRA is a novel method for parameter-efficient fine-tuning in large pre-trained language models (PLMs)
- It linearly allocates a different rank for each layer in an increasing manner and incorporates pruning throughout the training process
- Demonstrated superior performance on eight GLUE benchmarks compared to state-of-the-art metrics while maintaining the same number of trainable parameters
- Emphasizes the importance of adaptation in both input and output domains when transitioning between tasks, with a focus on co-adaptation of earlier layers
- Offers a simple yet effective solution for improving low-rank adaptation during fine-tuning processes, setting a new standard in parameter-efficient fine-tuning

## Summary

, , , , 

PRILoRA, short for Pruned and Rank-Increasing Low-Rank Adaptation, is a novel method that addresses the inefficiency of fine-tuning all model parameters in large pre-trained language models (PLMs). With the increasing complexity of downstream tasks, parameter-efficient fine-tuning has become a crucial area of research. Existing methods like Low-Rank Adaptation (LoRA) have shown promise by incorporating trainable rank decomposition matrices into target modules but often overlook the varying importance of each layer. In response to these challenges, PRILoRA linearly allocates a different rank for each layer in an increasing manner and incorporates pruning throughout the training process. This approach takes into account both the temporary magnitude of weights and the accumulated statistics of input to any given layer. Through extensive experiments on eight GLUE benchmarks, PRILoRA has demonstrated superior performance compared to state-of-the-art metrics while maintaining the same number of trainable parameters. The discussion surrounding PRILoRA emphasizes the need for adaptation in both input and output domains when transitioning between tasks. While top layers require more adaptation due to their proximity to the output, neglecting co-adaptation of earlier layers can hinder overall performance. The gradual increase in allocated resources implemented by PRILoRA proves to be a reasonable strategy for achieving optimal results. In conclusion, PRILoRA presents a simple yet effective solution for improving low-rank adaptation during fine-tuning processes. Its success across multiple seeds on various benchmarks showcases its efficiency in enhancing model performance while minimizing non-zero parameters. By setting a new standard in parameter-efficient fine-tuning, PRILoRA offers a promising avenue for optimizing PLMs for diverse downstream tasks such as question answering and text summarization.