# Language Models are Few-Shot Learners

**Authors:** Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse  
**arXiv:** [2005.14165](https://arxiv.org/abs/2005.14165)  
**Date:** 2020-05-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2005.14165v1/

## Key points

- Natural language processing (NLP) has made significant progress in recent years
- Pre-training models on large text corpora followed by fine-tuning for specific tasks is the current approach
- However, these models still require task-specific fine-tuning datasets consisting of thousands or tens of thousands of examples
- Humans can perform new language tasks from only a few examples or simple instructions, which current NLP systems struggle to do
- GPT-3 is an autoregressive language model with 175 billion parameters and tested its performance in the few-shot setting without any gradient updates or fine-tuning
- GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering and cloze tasks as well as tasks that require on-the-fly reasoning or domain adaptation such as unscrambling words, using a novel word in a sentence or performing 3 digit arithmetic.
- There are some datasets where GPT-3's few shot learning still struggles and faces methodological issues related to training on large web corpora.
- GPT-3 can generate samples of news articles that human evaluators have difficulty distinguishing from articles written by humans.
- Scaling up language models like GPT-3 greatly improves task agnostic few shot performance and sometimes even reaches competitiveness with prior state of the art fine tuning approaches.
- -The development of such models has significant implications for various industries such as healthcare, finance and education where natural language processing plays an essential role.

## Summary

The field of natural language processing (NLP) has seen significant progress in recent years with the development of pre-training models on large text corpora followed by fine-tuning for specific tasks. However, these models still require task-specific fine-tuning datasets consisting of thousands or tens of thousands of examples. In contrast, humans can perform new language tasks from only a few examples or simple instructions, which current NLP systems struggle to do. To address this gap, researchers have developed GPT-3, an autoregressive language model with 175 billion parameters - 10 times more than any previous non-sparse language model - and tested its performance in the few-shot setting without any gradient updates or fine-tuning. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering and cloze tasks as well as tasks that require on-the-fly reasoning or domain adaptation such as unscrambling words, using a novel word in a sentence or performing 3 digit arithmetic. However, there are some datasets where GPT-3's few shot learning still struggles and faces methodological issues related to training on large web corpora. One remarkable finding is that GPT-3 can generate samples of news articles that human evaluators have difficulty distinguishing from articles written by humans. This raises broader societal impacts and ethical concerns about the potential misuse of such technology. Overall, scaling up language models like GPT-3 greatly improves task agnostic few shot performance and sometimes even reaches competitiveness with prior state of the art fine tuning approaches. The development of such models has significant implications for various industries such as healthcare, finance and education where natural language processing plays an essential role.