# Scaling Instruction-Finetuned Language Models

**Authors:** Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao  
**arXiv:** [2210.11416](https://arxiv.org/abs/2210.11416)  
**Date:** 2022-10-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.11416v1/

## Key points

- Instruction finetuning improves performance and generalization of language models
- Three aspects focused on: scaling number of tasks, scaling model size, and finetuning on chain-of-thought data
- Results show significant improvement across various model classes, setups, and evaluation benchmarks
- Flan-PaLM 540B instruction-finetuned outperforms PALM 540B by a large margin (+9.4% on average)
- Flan-PaLM achieves state-of-the-art performance on several benchmarks
- Flan-T5 checkpoints demonstrate strong few-shot performance compared to larger models like PaLM 62B
- Instruction finetuning is a valuable method for improving pretrained language models with minimal computational cost

## Summary

In this paper, the authors explore the concept of instruction finetuning for language models and its impact on model performance and generalization to unseen tasks. They specifically focus on three aspects: scaling the number of tasks, scaling the model size, and finetuning on chain-of-thought data. The results show that instruction finetuning significantly improves performance across various model classes (PaLM, T5, U-PaLM), setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For example, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average) and achieves state-of-the-art performance on several benchmarks. Additionally, the authors release Flan-T5 checkpoints that demonstrate strong few-shot performance compared to larger models like PaLM 62B. The study concludes that instruction finetuning is a valuable method for improving the performance and usability of pretrained language models with minimal computational cost.