# CoCon: A Self-Supervised Approach for Controlled Text Generation

**Authors:** Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, Jie Fu  
**arXiv:** [2006.03535](https://arxiv.org/abs/2006.03535)  
**Date:** 2020-06-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2006.03535v1/

## Key points

- Paper presents a method called CoCon for controlling output of pretrained Transformer-based language models at a fine-grained level
- Previous studies focused on high-level attributes, lacking control over word- and phrase-level content
- CoCon is a Content-Conditioner that allows targeted content input to guide LM's output
- Self-supervised approach where CoCon learns to complete partially observed text sequences by conditioning with withheld content inputs
- Experiments show CoCon effectively incorporates target content and controls high-level text attributes in a zero-shot manner
- Potential applications include personalized text generation, data augmentation, and controlled paraphrasing
- Authors: Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, Jie Fu

## Summary

The paper titled "CoCon: A Self-Supervised Approach for Controlled Text Generation" presents a method to control the output of pretrained Transformer-based language models (LMs) at a fine-grained level. Previous studies have focused on controlling high-level attributes like sentiment and topic; however, there is still a lack of precise control over the content of generated text at the word- and phrase-level. To address this limitation, the authors propose the Content-Conditioner (CoCon), which allows for targeted content input to guide the LM's output. In their self-supervised approach, CoCon learns to complete partially observed text sequences by conditioning with withheld content inputs. Experiments demonstrate that CoCon can effectively incorporate target content into generated texts and control high-level text attributes in a zero-shot manner. This research has potential to enhance natural language generation capabilities by providing more control over specific aspects of generated text. By enabling fine-grained control over word- and phrase-level content, CoCon opens up possibilities for applications such as personalized text generation, data augmentation, and controlled paraphrasing. The authors of this paper are Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, and Jie Fu; their work contributes to advancing the field of natural language processing and showcases the potential of self-supervised approaches for improving language model outputs.