# Federated Fine-tuning of Billion-Sized Language Models across Mobile Devices

**Authors:** Mengwei Xu, Yaozong Wu, Dongqi Cai, Xiang Li, Shangguang Wang  
**arXiv:** [2308.13894](https://arxiv.org/abs/2308.13894)  
**Date:** 2023-08-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.13894v1/

## Key points

- Large Language Models (LLMs) have revolutionized mobile intelligence
- Federated Learning (FL) is key for fine-tuning LLMs for mobile tasks
- FedLLM preserves user data privacy but faces challenges like high memory consumption and slow convergence
- FwdLLM introduces innovative FL protocol using BP-free training methods for improved memory and time efficiency
- FwdLLM focuses on combining BP-free training with parameter-efficient methods, optimal computational load allocation, and selective sampling of perturbed predictions
- Benefits of FwdLLM include faster convergence, reduced memory footprint, and enabling federated learning of billion-parameter LLMs on COTS mobile devices

## Summary

, , , , 

In the realm of mobile intelligence, Large Language Models (LLMs) have revolutionized the landscape, with Federated Learning (FL) being a key method to fine-tune these models for downstream mobile tasks. This approach, known as FedLLM, has been instrumental in preserving user data privacy. However, challenges persist in integrating these vast models with mobile devices, such as high memory consumption and slow model convergence. To address these issues, a new innovative FL protocol called FwdLLM has been introduced. FwdLLM leverages backpropagation (BP)-free training methods, requiring devices to execute "perturbed inferences" instead. This unique approach results in significantly improved memory efficiency and time efficiency, aided by mobile NPUs and an expanded array of participant devices. FwdLLM focuses on three key designs: combining BP-free training with parameter-efficient methods to scale to the LLM era; systematically allocating computational loads across devices for optimal convergence speed and accuracy; and selectively sampling perturbed predictions crucial for model convergence. Extensive experiments involving five LLMs and three NLP tasks demonstrate the remarkable advantages of FwdLLM over traditional methods. These benefits include up to three orders of magnitude faster convergence and a substantial 14.6x reduction in memory footprint. Notably, FwdLLM opens up possibilities for federated learning of billion-parameter LLMs like LLaMA on commercial off-the-shelf (COTS) mobile devices – a previously unattained achievement. The research conducted by Mengwei Xu, Yaozong Wu, Dongqi Cai, Xiang Li, and Shangguang Wang showcases how FwdLLM is poised to enhance the efficiency of FedLLM and pave the way for more advanced applications of large language models on mobile platforms.