# Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes

**Authors:** Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, Shuiguang Deng  
**arXiv:** [2312.06353](https://arxiv.org/abs/2312.06353)  
**Date:** 2023-12-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.06353v5/

## Key points

- Authors address the need for fine-tuning pre-trained large language models (LLMs) to enhance responsiveness to natural language instructions
- Proposed approach called FedKSeed leverages federated learning for fine-tuning LLMs using data from end devices while ensuring data privacy
- Introduces zeroth-order optimization with a finite set of random seeds to reduce communication costs involved in full-parameter tuning in a federated setting
- Breakthrough enables federated full-parameter tuning of billion-sized LLMs directly on devices by drastically reducing transmission requirements
- Strategy within FedKSeed facilitates probability-differentiated seed sampling, prioritizing perturbations with greater impact on model accuracy
- Extensive experiments demonstrate that FedKSeed surpasses existing federated LLM fine-tuning methods in terms of communication efficiency and generalization to new tasks

## Summary

In their paper titled "Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes," authors Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng address the need for fine-tuning pre-trained large language models (LLMs) to enhance their responsiveness to natural language instructions. They propose a novel approach called FedKSeed that leverages federated learning to fine-tune LLMs using data from end devices while ensuring data privacy. The existing federated fine-tuning methods for LLMs typically rely on parameter-efficient techniques, which may not fully exploit the potential performance gains achievable through full-parameter tuning. However, conducting full-parameter tuning in a federated setting poses challenges due to the significant communication costs involved. To overcome this obstacle, FedKSeed introduces zeroth-order optimization with a finite set of random seeds. This drastically reduces the transmission requirements between the server and clients to just a few random seeds and scalar gradients totaling only a few thousand bytes. This breakthrough enables the possibility of federated full-parameter tuning of billion-sized LLMs directly on devices. Furthermore, the authors develop a strategy within FedKSeed that facilitates probability-differentiated seed sampling, prioritizing perturbations that have a greater impact on model accuracy. Through extensive experiments across six scenarios involving various LLMs, datasets, and data partitions, they demonstrate that their approach surpasses existing federated LLM fine-tuning methods in terms of both communication efficiency and generalization to new tasks. The research conducted by Qin et al., accepted at ICML 2024 and spanning 25 pages with 14 figures and 7 tables showcases the effectiveness of FedKSeed in optimizing large language models efficiently while maintaining data privacy. The authors provide access to their codes on GitHub for further exploration and implementation in real-world applications.