# BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

**Authors:** Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg  
**arXiv:** [2106.10199](https://arxiv.org/abs/2106.10199)  
**Date:** 2021-06-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.10199v5/

## Key points

- BitFit is a sparse-finetuning method that modifies only the bias-terms of a model or a subset of them.
- BitFit can be as effective, if not better, than fine-tuning the entire pre-trained BERT model for small-to-medium training data.
- For larger datasets, BitFit remains competitive with other sparse fine-tuning methods available.
- The study challenges traditional assumptions about finetuning in natural language processing tasks and offers insights into efficient strategies for adapting transformer-based language models.
- Existing approaches like "Adapters" and "Diff-Pruning" have successfully introduced task-specific capabilities without significantly altering the original model architecture or parameters.

## Summary

, , , , 

In the study titled "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models," authors Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg introduce BitFit, a sparse-finetuning method that modifies only the bias-terms of a model or a subset of them. The research demonstrates that BitFit can be just as effective, if not better, than fine-tuning the entire pre-trained BERT model when dealing with small-to-medium training data. For larger datasets, BitFit remains competitive with other sparse fine-tuning methods available. This study offers valuable insights into efficient strategies for adapting transformer-based language models and challenges traditional assumptions about the nature of finetuning in natural language processing tasks. It also discusses existing approaches such as "Adapters" by Houlsby et al. (2019) and "Diff-Pruning" by Guo et al. (2020), which have successfully introduced task-specific capabilities without significantly altering the original model architecture or parameters. Overall, this research highlights the importance of leveraging existing knowledge within pre-trained models for effective adaptation to new tasks and has practical implications for improving model performance and efficiency in natural language processing applications.