# Low-rank Attention Side-Tuning for Parameter-Efficient Fine-Tuning

**Authors:** Ningyuan Tang, Minghao Fu, Ke Zhu, Jianxin Wu  
**arXiv:** [2402.04009](https://arxiv.org/abs/2402.04009)  
**Date:** 2024-02-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.04009v1/

## Key points

- Paper introduces Low-rank Attention Side-Tuning (LAST) for fine-tuning large pretrained models
- LAST disentangles trainable module from pretrained model by freezing parameters and outputs
- Trains a side-network with low-rank self-attention modules to focus on task-specific knowledge
- Highly parallel across multiple optimization objectives, efficient in downstream task adaptation and hyperparameter optimization
- Outperforms previous state-of-the-art methods on visual adaptation tasks like VTAB-1K
- Achieves higher accuracy, consumes 30% less GPU memory, and requires 60% less training time compared to existing PEFT methods

## Summary

The paper "Low-rank Attention Side-Tuning for Parameter-Efficient Fine-Tuning" by Ningyuan Tang, Minghao Fu, Ke Zhu, and Jianxin Wu introduces a novel approach to fine-tuning large pretrained models for downstream tasks. The authors propose Low-rank Attention Side-Tuning (LAST), a method that disentangles the trainable module from the pretrained model by freezing not only parameters but also outputs of the pretrained network. LAST trains a side-network consisting of low-rank self-attention modules and leverages the frozen feature extractor capabilities of the pretrained model to focus on learning task-specific knowledge. This allows LAST to be highly parallel across multiple optimization objectives, making it efficient in downstream task adaptation and hyperparameter optimization. The authors demonstrate that LAST outperforms previous state-of-the-art methods on visual adaptation tasks such as VTAB-1K, achieving significantly higher accuracy while consuming roughly 30% less GPU memory and requiring 60% less training time compared to existing PEFT methods. Overall, LAST represents a promising advancement in parameter-efficient fine-tuning techniques with improved performance and efficiency in adapting pretrained models to specific downstream tasks.