# Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

**Authors:** Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel  
**arXiv:** [2205.05638](https://arxiv.org/abs/2205.05638)  
**Date:** 2022-05-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.05638v2/

## Key points

- Comparison of two methods for adapting pre-trained language models to new tasks: Few-shot in-context learning (ICL) and Parameter-efficient fine-tuning (PEFT)
- ICL processes all training examples for each prediction, leading to high computational costs
- PEFT focuses on training a small set of parameters for task performance with lower costs
- Introduction of a new PEFT method called (IA)$^3$ using learned vectors to scale activations and improve performance without adding many parameters
- Proposal of T-Few, a simple recipe based on the T0 model that can be applied to new tasks without task-specific tuning or modifications
- Rigorous comparison showing that PEFT offers better accuracy and significantly lower computational costs compared to ICL
- Validation of T-Few on unseen tasks by applying it to the RAFT benchmark, achieving super-human performance and outperforming existing methods by 6% absolute
- Highlighting the advantages of PEFT over ICL in terms of efficiency and performance when adapting pre-trained language models to new tasks
- Availability of code used in experiments for further exploration and validation

## Summary

This paper compares the effectiveness of two methods for adapting pre-trained language models to new tasks: Few-shot in-context learning (ICL) and Parameter-efficient fine-tuning (PEFT). While ICL processes all training examples for each prediction, leading to high computational costs, PEFT focuses on training a small set of parameters for task performance with lower costs. The authors introduce a new PEFT method called (IA)$^3$, which uses learned vectors to scale activations and improve performance without adding many parameters. They also propose T-Few, a simple recipe based on the T0 model that can be applied to new tasks without task-specific tuning or modifications. Through rigorous comparison, the authors demonstrate that PEFT offers better accuracy and significantly lower computational costs compared to ICL. They validate the effectiveness of T-Few on unseen tasks by applying it to the RAFT benchmark, achieving super-human performance and outperforming existing methods by 6% absolute. Overall, this study highlights the advantages of PEFT over ICL in terms of efficiency and performance when adapting pre-trained language models to new tasks. The code used in their experiments is publicly available for further exploration and validation.