# The Power of Scale for Parameter-Efficient Prompt Tuning

**Authors:** Brian Lester, Rami Al-Rfou, Noah Constant  
**arXiv:** [2104.08691](https://arxiv.org/abs/2104.08691)  
**Date:** 2021-04-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.08691v2/

## Key points

- Prompt tuning is a method to enhance frozen language models' performance in downstream tasks
- Soft prompts are learned through backpropagation, incorporating signals from labeled examples
- End-to-end learned prompt tuning surpasses GPT-3's few-shot learning capabilities significantly
- Prompt tuning becomes more competitive with larger-scale models, offering a cost-effective solution
- It is a simplified version of prefix tuning and enhances robustness in domain transfer scenarios
- Prompt tuning optimizes frozen language models efficiently and economically as model sizes grow exponentially

## Summary

In their paper titled "The Power of Scale for Parameter-Efficient Prompt Tuning," authors Brian Lester, Rami Al-Rfou, and Noah Constant delve into the concept of prompt tuning as a method to enhance the performance of frozen language models in specific downstream tasks. Prompt tuning involves learning "soft prompts" through backpropagation, allowing them to incorporate signals from numerous labeled examples. The researchers demonstrate that their end-to-end learned approach surpasses the few-shot learning capabilities of GPT-3 by a significant margin. Through experiments with model size variations using T5, the authors illustrate that prompt tuning becomes increasingly competitive with larger-scale models. This finding is particularly noteworthy considering the high costs associated with sharing and serving large models. By enabling a single frozen model to cater to multiple downstream tasks, prompt tuning offers a cost-effective solution. The study also positions prompt tuning as a simplified version of prefix tuning, introduced by Li and Liang in 2021. A comparison between these approaches and other similar methods is provided for comprehensive understanding. Additionally, the authors highlight that conditioning a frozen model with soft prompts enhances robustness in domain transfer scenarios compared to full model tuning. Overall, this research sheds light on how prompt tuning can optimize frozen language models for various tasks efficiently and economically, especially as model sizes continue to grow exponentially in complexity and scale. Prompt tuning has proven to be an effective method for enhancing the performance of frozen language models in specific downstream tasks. By incorporating soft prompts through backpropagation and utilizing end-to-end learning techniques, it surpasses traditional discrete text prompts used by models like GPT-3 in few-shot learning capabilities. Furthermore, experiments with different model sizes demonstrate the increasing competitiveness of prompt tuning, making it a viable alternative to full model tuning. This cost-effective solution is particularly valuable as model sizes continue to grow exponentially, and prompt tuning offers a simplified version of prefix tuning that enhances robustness in domain transfer scenarios. In conclusion, this research highlights the power of scale for parameter-efficient prompt tuning and its potential to optimize frozen language models for various tasks efficiently and economically.