# ResLoRA: Identity Residual Mapping in Low-Rank Adaption

**Authors:** Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang  
**arXiv:** [2402.18039](https://arxiv.org/abs/2402.18039)  
**Date:** 2024-02-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.18039v1/

## Key points

- Low-rank adaptation (LoRA) is a popular technique for fine-tuning large language models (LLMs) in the field of parameter-efficient fine-tuning (PEFT).
- ResLoRA, a novel framework, improves upon LoRA by incorporating residual paths during training and using merging approaches to eliminate these paths during inference.
- ResLoRA achieves superior results in fewer training steps without introducing extra trainable parameters or inference cost compared to traditional LoRA.
- Experiments on tasks like Natural Language Generation (NLG), Natural Language Understanding (NLU), and text-to-image tasks have shown the effectiveness of ResLoRA.
- ResLoRA is the first work to combine residual paths with LoRA, showcasing an innovative approach towards enhancing parameter-efficient fine-tuning methods.

## Summary

, , , , 

In the field of parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs). However, updating the weights of LoRA blocks poses a significant challenge due to the long calculation path in the original model. To address this issue, a novel framework called ResLoRA has been proposed. This improved version of LoRA incorporates residual paths during training and employs merging approaches to eliminate these additional paths during inference. The key advantage of ResLoRA is its ability to achieve superior results in fewer training steps without introducing any extra trainable parameters or inference cost compared to traditional LoRA. The experiments conducted on various tasks such as Natural Language Generation (NLG), Natural Language Understanding (NLU), and text-to-image tasks have demonstrated the effectiveness of ResLoRA. Notably, ResLoRA stands out as the first work to combine residual paths with LoRA, showcasing its innovative approach towards enhancing parameter-efficient fine-tuning methods. The research paper titled "ResLoRA: Identity Residual Mapping in Low-Rank Adaption" authored by Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang provides detailed insights into the development and implementation of ResLoRA. Interested readers can access the code for this method on GitHub at https://github.com/microsoft/LMOps/tree/main/reslora.