Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

AI-generated keywords: Prompt-based Learning NLP Pre-training Template Search Answer Search

AI-generated Key Points

  • Natural Language Processing (NLP) has a new paradigm called "prompt-based learning"
  • Prompt-based learning uses language models that directly model the probability of text
  • Original input x is modified using a template into a textual string prompt x', which has some unfilled slots
  • Prompt-based learning allows for pre-training on massive amounts of raw text and can perform few-shot or even zero-shot learning by adapting to new scenarios with few or no labeled data
  • Most tasks regarding prompt-based learning revolve around text classification and factual probing because both template engineering and answer engineering are relatively easy to conduct, and experiments are computationally inexpensive for these tasks
  • Current research mainly focuses on template search instead of answer search
  • The authors hope that their survey will help researchers better understand the paradigm of prompt-based learning and its core challenges so that more scientifically meaningful advances can be made in this field.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig

Website: http://pretrain.nlpedia.ai/
License: CC BY-SA 4.0

Abstract: This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g.the choice of pre-trained models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website http://pretrain.nlpedia.ai/ including constantly-updated survey, and paperlist.

Submitted to arXiv on 28 Jul. 2021

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2107.13586v1

The field of natural language processing (NLP) has seen a new paradigm in recent years called "prompt-based learning". This approach is different from traditional supervised learning, where a model is trained to predict an output y given an input x. Instead, prompt-based learning uses language models that directly model the probability of text. To perform prediction tasks, the original input x is modified using a template into a textual string prompt x', which has some unfilled slots. The language model then probabilistically fills in the unfilled information to obtain a final string x, from which the final output y can be derived. Prompt-based learning has several advantages over traditional supervised learning. It allows for pre-training on massive amounts of raw text and can perform few-shot or even zero-shot learning by adapting to new scenarios with few or no labeled data. In this paper, researchers introduce the basics of this promising paradigm and organize existing work along several dimensions such as pre-trained models, prompts, and tuning strategies. The authors also provide insights into trends in prompt-based research through trend analysis and timeline summaries of existing research papers. They note that most tasks regarding prompt-based learning revolve around text classification and factual probing because both template engineering and answer engineering are relatively easy to conduct, and experiments are computationally inexpensive for these tasks. While both prompt and answer search are important tools for taking advantage of pre-trained language models for many tasks, current research mainly focuses on template search instead of answer search. The authors suggest that this may be due to gold references being directly usable as answers for conditional generation tasks like summarization or translation. For classification tasks, label words are relatively easy to select using domain knowledge. Overall, the authors hope that their survey will help researchers better understand the paradigm of prompt-based learning and its core challenges so that more scientifically meaningful advances can be made in this field. They also highlight commonalities and differences between different paradigms in NLP research, potentially providing a catalyst to inspire work towards the next paradigm shift.
Created on 09 Apr. 2023

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.