# Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

**Authors:** Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig  
**arXiv:** [2107.13586](https://arxiv.org/abs/2107.13586)  
**Date:** 2021-07-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2107.13586v1/

## Key points

- Natural Language Processing (NLP) has a new paradigm called "prompt-based learning"
- Prompt-based learning uses language models that directly model the probability of text
- Original input x is modified using a template into a textual string prompt x', which has some unfilled slots
- Prompt-based learning allows for pre-training on massive amounts of raw text and can perform few-shot or even zero-shot learning by adapting to new scenarios with few or no labeled data
- Most tasks regarding prompt-based learning revolve around text classification and factual probing because both template engineering and answer engineering are relatively easy to conduct, and experiments are computationally inexpensive for these tasks
- Current research mainly focuses on template search instead of answer search
- The authors hope that their survey will help researchers better understand the paradigm of prompt-based learning and its core challenges so that more scientifically meaningful advances can be made in this field.

## Summary

The field of natural language processing (NLP) has seen a new paradigm in recent years called "prompt-based learning". This approach is different from traditional supervised learning, where a model is trained to predict an output y given an input x. Instead, prompt-based learning uses language models that directly model the probability of text. To perform prediction tasks, the original input x is modified using a template into a textual string prompt x', which has some unfilled slots. The language model then probabilistically fills in the unfilled information to obtain a final string x, from which the final output y can be derived. Prompt-based learning has several advantages over traditional supervised learning. It allows for pre-training on massive amounts of raw text and can perform few-shot or even zero-shot learning by adapting to new scenarios with few or no labeled data. In this paper, researchers introduce the basics of this promising paradigm and organize existing work along several dimensions such as pre-trained models, prompts, and tuning strategies. The authors also provide insights into trends in prompt-based research through trend analysis and timeline summaries of existing research papers. They note that most tasks regarding prompt-based learning revolve around text classification and factual probing because both template engineering and answer engineering are relatively easy to conduct, and experiments are computationally inexpensive for these tasks. While both prompt and answer search are important tools for taking advantage of pre-trained language models for many tasks, current research mainly focuses on template search instead of answer search. The authors suggest that this may be due to gold references being directly usable as answers for conditional generation tasks like summarization or translation. For classification tasks, label words are relatively easy to select using domain knowledge. Overall, the authors hope that their survey will help researchers better understand the paradigm of prompt-based learning and its core challenges so that more scientifically meaningful advances can be made in this field. They also highlight commonalities and differences between different paradigms in NLP research, potentially providing a catalyst to inspire work towards the next paradigm shift.