# Run LoRA Run: Faster and Lighter LoRA Implementations

**Authors:** Daria Cherniuk, Aleksandr Mikhalev, Ivan Oseledets  
**arXiv:** [2312.03415](https://arxiv.org/abs/2312.03415)  
**Date:** 2023-12-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.03415v1/

## Key points

- LoRA is a technique used in neural networks to reduce the number of trainable parameters
- Low-rank adapters are introduced to linear layers for fine-tuning and full training
- The RunLoRA framework offers efficient implementations of LoRA, improving training and fine-tuning speed
- Computation of LoRA operations is optimized based on factors such as layer dimensions and LoRA rank
- Forward and backward computation graphs are chosen based on FLOPs and time estimations for faster training without sacrificing accuracy
- RunLoRA achieves a speedup of up to 17% in experiments on Llama family models
- Multiple variants of forward and backward passes are included in the implementation with different bracket placements and operation reorderings
- Calculations for the backward pass involve tensors such as dA, dB, and dX, which can be performed in several ways due to associativity of matrix multiplications.

## Summary

LoRA is a technique used in neural networks to reduce the number of trainable parameters by introducing low-rank adapters to linear layers. These adapters are used for both fine-tuning and full training. In this paper, the authors present the RunLoRA framework, which offers efficient implementations of LoRA that significantly improve the speed of neural network training and fine-tuning using low-rank adapters. The proposed implementation optimizes the computation of LoRA operations based on various factors such as the dimensions of corresponding linear layers, layer input dimensions, and LoRA rank. By choosing the best forward and backward computation graph based on FLOPs (floating point operations) and time estimations, the framework achieves faster training without sacrificing accuracy. The authors conducted experiments to evaluate the performance of RunLoRA on Llama family models. The results show a speedup of up to 17%, demonstrating its effectiveness in improving training efficiency. In terms of problem setting and methodology, the default forward pass of LoRA involves computing LoRA(X) = XW + (XA)B. However, many researchers avoid this approach due to assumptions about large weights W and undesired same-size matrix AB formation. The current implementation includes multiple variants of forward and backward passes, with different bracket placements and operation reorderings. The backward pass requires calculating tensors such as dA = X⊤dY B⊤, dB = A⊤X⊤dY, and dX = dY W ⊤ + dY B⊤A⊤. There are several ways to perform these calculations due to associativity of matrix multiplications. Overall, this paper introduces an efficient framework for implementing LoRA called RunLoRA. It improves training speed by optimizing computations based on various factors while maintaining accuracy. The experimental results demonstrate its effectiveness in speeding up neural network training using low-rank adapters.