# QLoRA: Efficient Finetuning of Quantized LLMs

**Authors:** Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer  
**arXiv:** [2305.14314](https://arxiv.org/abs/2305.14314)  
**Date:** 2023-05-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14314v1/

## Key points

- QLoRA is an efficient finetuning approach that enables the finetuning of a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance
- Gradients are backpropagated through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA)
- Innovations to save memory without sacrificing performance include the use of 4-bit NormalFloat (NF4), double quantization, and paged optimizers
- QLoRA was used to finetune more than 1,000 models and provide a detailed analysis of instruction following and chatbot performance across eight instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning
- Results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results even when using smaller models than the previous state-of-the-art
- Qualitative analysis shows that their best model family named Guanaco outperforms all previously released models on the Vicuna benchmark reaching 99.3% of ChatGPT's performance level while only requiring 24 hours of finetuning on a single GPU
- They also provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation
- The authors release all their models and code including CUDA kernels for 4-bit training which will enable further exploration in this area

## Summary

The paper presents QLoRA, an efficient finetuning approach that enables the finetuning of a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. The authors achieve this by backpropagating gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA). They introduce several innovations to save memory without sacrificing performance, including the use of 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights, double quantization to reduce the average memory footprint by quantizing the quantization constants, and paged optimizers to manage memory spikes. The authors use QLoRA to finetune more than 1,000 models and provide a detailed analysis of instruction following and chatbot performance across eight instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g., 33B and 65B parameter models). Their results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results even when using smaller models than the previous state-of-the-art. In addition to quantitative analysis, the authors perform qualitative analysis in two sections. First, they analyze Elo ratings for a tournament between models where models compete to generate the best response for a prompt judged by human raters or GPT-4. Overall, their best model family named Guanaco outperforms all previously released models on the Vicuna benchmark reaching 99.3% of ChatGPT's performance level while only requiring 24 hours of finetuning on a single GPU. Second, they provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. They also find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. The authors acknowledge some limitations of their work including the lack of analysis on non-English languages and the need for more research on how to improve the quality of instruction following models. Overall their approach presents a promising direction for efficient finetuning of large language models while maintaining high performance. The authors release all their models and code including CUDA kernels for 4-bit training which will enable further exploration in this area.