# Full Parameter Fine-tuning for Large Language Models with Limited Resources

**Authors:** Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, Xipeng Qiu  
**arXiv:** [2306.09782](https://arxiv.org/abs/2306.09782)  
**Date:** 2023-06-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.09782v1/

## Key points

- Large Language Models (LLMs) have transformed Natural Language Processing (NLP), but their training requires massive GPU resources, making it difficult for researchers with limited resources to participate.
- Existing approaches have focused on parameter-efficient fine-tuning, which tunes or adds a small number of parameters.
- Tuning the full parameters of LLMs with limited resources remains a significant challenge.
- Kai Lv et al. propose a new optimizer called LOw-Memory Optimization (LOMO) that fuses gradient computation and parameter updates in one step to reduce memory usage.
- By integrating LOMO with existing memory-saving techniques, the authors were able to reduce memory usage to 10.8% compared to the standard approach (DeepSpeed solution).
- This reduction in memory usage enables the full parameter fine-tuning of a 65B model on a single machine with eight RTX 3090 GPUs, each having 24GB memory.
- The proposed approach significantly lowers the threshold for LLMs training and encourages greater participation from researchers, benefiting both academia and society.
- The authors' work provides an efficient method for training large language models without requiring extensive computational resources.
- This will enable more researchers worldwide to access NLP research and development activities more easily and quickly progress in this field.
- Innovative optimization techniques can help overcome resource constraints and facilitate breakthroughs in NLP research.

## Summary

Large Language Models (LLMs) have transformed Natural Language Processing (NLP), but their training requires massive GPU resources, making it difficult for researchers with limited resources to participate. To address this challenge, existing approaches have focused on parameter-efficient fine-tuning, which tunes or adds a small number of parameters. However, tuning the full parameters of LLMs with limited resources remains a significant challenge. In this work titled "Full Parameter Fine-tuning for Large Language Models with Limited Resources," Kai Lv et al. propose a new optimizer called LOw-Memory Optimization (LOMO) that fuses gradient computation and parameter updates in one step to reduce memory usage. By integrating LOMO with existing memory-saving techniques, the authors were able to reduce memory usage to 10.8% compared to the standard approach (DeepSpeed solution). This reduction in memory usage enables the full parameter fine-tuning of a 65B model on a single machine with eight RTX 3090 GPUs, each having 24GB memory. The proposed approach significantly lowers the threshold for LLMs training and encourages greater participation from researchers, benefiting both academia and society. The authors' work is crucial because it provides an efficient method for training large language models without requiring extensive computational resources. This will enable more researchers worldwide to access NLP research and development activities more easily and quickly progress in this field. Overall, this study's findings demonstrate how innovative optimization techniques can help overcome resource constraints and facilitate breakthroughs in NLP research.