# Language Models are Few-Shot Learners

**Authors:** Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse  
**arXiv:** [2005.14165](https://arxiv.org/abs/2005.14165)  
**Date:** 2020-05-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2005.14165v4/

## Key points

- Pre-training language models on a large corpus of text and fine-tuning them on specific tasks can improve NLP performance.
- Humans can perform new language tasks with few examples, while current NLP systems struggle.
- Scaling up language models can enhance task-agnostic few-shot performance.
- GPT-3, an autoregressive language model with 175 billion parameters, achieved strong performance without fine-tuning.
- GPT-3 performed well on various NLP datasets, including translation, question answering, cloze tasks, reasoning, and domain adaptation.
- Some datasets still pose challenges for GPT-3's few-shot learning.
- GPT-3 is capable of generating news articles that are difficult to distinguish from human-written ones.
- The societal impacts and implications of advanced language models like GPT-3 are discussed in the study.

## Summary

In recent work, it has been shown that pre-training language models on a large corpus of text and then fine-tuning them on specific tasks can lead to significant improvements in natural language processing (NLP) performance. However, this method still requires task-specific fine-tuning datasets with thousands or tens of thousands of examples. In contrast, humans are able to perform new language tasks with only a few examples or simple instructions, which current NLP systems struggle to do. This study demonstrates that scaling up language models can greatly enhance their task-agnostic few-shot performance, sometimes even surpassing the state-of-the-art fine-tuning approaches. The researchers trained GPT-3, an autoregressive language model with 175 billion parameters - ten times more than any previous non-sparse language model - and evaluated its performance in the few-shot setting. Importantly, GPT-3 was applied without any gradient updates or fine-tuning and relied solely on text interaction for task specification and demonstration. The results show that GPT-3 achieves strong performance on various NLP datasets, including translation, question answering, cloze tasks as well as tasks requiring on the fly reasoning or domain adaptation such as unscrambling words using novel words in sentences or performing 3 digit arithmetic. However there are still some datasets where GPT 3's few shot learning struggles and others where methodological issues related to training on large web corpora arise. Additionally the study found that GPT 3 is capable of generating news article samples that human evaluators have difficulty distinguishing from articles written by humans. The broader societal impacts of this finding and the implications of GPT 3 are discussed. Overall this research highlights the potential of scaling up language models like GPT 3 to improve task agnostic few shot learning in NLP tasks. However challenges remain in certain datasets and methodological considerations when training on large web corpora. The study also raises important questions about the societal implications of advanced language models like GPT 3.