# Accurate LoRA-Finetuning Quantization of LLMs via Information Retention

**Authors:** Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, Michele Magno  
**arXiv:** [2402.05445](https://arxiv.org/abs/2402.05445)  
**Date:** 2024-02-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.05445v2/

## Key points

- Quantization of Large Language Models (LLMs) using LoRA-finetuning is a key area of research in machine learning.
- Existing methods have limitations in maintaining performance when quantizing LLMs with LoRA finetuning.
- IR-QLoRA, introduced by Haotong Qin, Xudong Ma, and team, focuses on information retention to enhance accuracy of quantized LLMs with LoRA.
- IR-QLoRA leverages Statistics-based Information Calibration Quantization and Finetuning-based Information Elastic Connection for unified information processing.
- Extensive experiments show that IR-QLoRA significantly improves accuracy across various LLaMA and LLaMA2 model families under 2-4 bit-width configurations.
- For example, a 4-bit LLaMA-7B model achieved a 1.4% enhancement in Mean Model Log-Likelihood Uncertainty (MMLU) compared to state-of-the-art methods with minimal increase in time consumption.
- IR-QLoRA is versatile and compatible with different frameworks like NormalFloat and Integer quantization techniques while consistently delivering enhanced accuracy outcomes.
- Researchers can access the code implementation of IR-QLoRA at https://github.com/htqin/ir-qlora.

## Summary

, , , , 

In the realm of machine learning, the quantization of Large Language Models (LLMs) using LoRA-finetuning has been a topic of extensive research. The goal is to create accurate yet compact LLMs that can be deployed on hardware with limited resources. However, existing methods have shown limitations as they often lead to severe degradation in the performance of quantized LLMs, failing to fully leverage the benefits of LoRA finetuning. To address this challenge, a team of researchers led by Haotong Qin, Xudong Ma, and their colleagues have introduced a groundbreaking approach known as IR-QLoRA. This novel technique aims to enhance the accuracy of quantized LLMs with LoRA by focusing on information retention. IR-QLoRA leverages two key technologies centered around unified information processing: Statistics-based Information Calibration Quantization and Finetuning-based Information Elastic Connection. Through Statistics-based Information Calibration Quantization, this method enables the quantized parameters of LLMs to accurately retain their original information during the quantization process. Meanwhile, Finetuning-based Information Elastic Connection empowers LoRA to undergo elastic representation transformation that incorporates diverse information sources. Extensive experiments conducted by the research team demonstrate the effectiveness of IR-QLoRA in significantly improving accuracy across various LLaMA and LLaMA2 model families under 2-4 bit-width configurations. For instance, a 4-bit LLaMA-7B model achieved a remarkable 1.4% enhancement in Mean Model Log-Likelihood Uncertainty (MMLU) compared to state-of-the-art methods. These performance gains were achieved with only a minimal 0.31% increase in time consumption, highlighting the efficiency of IR-QLoRA. Furthermore, it is emphasized that IR-QLoRA exhibits excellent versatility and compatibility with different frameworks such as NormalFloat and Integer quantization techniques while consistently delivering enhanced accuracy outcomes. Researchers interested in exploring this innovative approach further can access the code implementation at https://github.com/htqin/ir-qlora. In conclusion, the introduction of IR-QLoRA represents a significant advancement in the field of quantized LLMs with LoRA finetuning, offering a promising solution for achieving highly accurate models while optimizing resource utilization and computational efficiency.