# A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

**Authors:** Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, Dawei Song  
**arXiv:** [2201.05337](https://arxiv.org/abs/2201.05337)  
**Date:** 2022-01-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.05337v1/

## Key points

- Controllable Text Generation (CTG) is a rapidly growing area in natural language generation (NLG)
- Transformer-based Pre-trained Language Models (PLMs) have emerged as a new paradigm in NLG
- Controllability of PLMs needs to be guaranteed due to the lower interpretability of deep neural networks
- Researchers have focused on controllable text generation using transformer-based PLMs
- Deep learning-based methods, such as GANs and Energy-based Models, have shown potential in text generation
- Large-scale pre-trained Language Models (PLMs), such as BERT, RoBERTa, GPT, T5, and mBART, have become a new paradigm in NLP since 2018
- PLMs can generate high-quality texts with fine-tuning for downstream tasks and specific constraints without external domain knowledge
- This paper presents a systematic critical review of common tasks, main approaches, and evaluation methods in CTG using PLM-based models
- Improving interpretability and controllability of PLMs is a hot research topic in generating text
- The paper focuses on PLM-based methods as they are becoming mainstream in CTG research
- The survey aims to help researchers understand the landscape and cutting-edge methods in PLM-based CTG.

## Summary

Controllable Text Generation (CTG) is a rapidly growing area in natural language generation (NLG) that aims to develop advanced text generation technologies that are more natural and better suited for practical applications. In recent years, transformer-based Pre-trained Language Models (PLMs) have emerged as a new paradigm in NLG, allowing for the generation of diverse and fluent text. However, the controllability of these methods needs to be guaranteed due to the lower interpretability of deep neural networks. To address this challenge, researchers have focused on controllable text generation using transformer-based PLMs. Various approaches have been proposed targeting different CTG tasks that require different types of controlled constraints. Deep learning-based methods, such as Generative Adversarial Networks (GANs) and Energy-based Models, have shown potential in text generation by learning low-dimensional dense vectors that represent linguistic features. However, these methods heavily rely on large-scale datasets, posing challenges for supervised and cross-domain text generation tasks. Large-scale pre-trained Language Models (PLMs), such as BERT, RoBERTa, GPT, T5, and mBART, have become a new paradigm in NLP since 2018. These models leverage unsupervised learning based on the Transformer structure to learn semantic and syntactical knowledge from large corpora. They can generate high-quality texts with fine-tuning for downstream tasks. Moreover, PLMs can generate text with specific constraints without external domain knowledge. This paper presents a systematic critical review of common tasks, main approaches, and evaluation methods in this area. It also discusses the challenges faced by the field and suggests promising future directions. However, PLMs are still black boxes with limited interpretability and controllability. Improving their interpretability and controllability has become a hot research topic in generating text using PLM-based models. This paper focuses on PLM-based methods as they are becoming mainstream in CTG research. It provides a comprehensive review of the current literature, including representative application tasks, main approaches, and evaluation methodologies. The paper also discusses future research directions. Overall, this survey aims to help researchers quickly understand the landscape and cutting-edge methods in PLM-based CTG.