# Typical Decoding for Natural Language Generation

**Authors:** Clara Meister, Tiago Pimentel, Gian Wiher, Ryan Cotterell  
**arXiv:** [2202.00666](https://arxiv.org/abs/2202.00666)  
**Date:** 2022-02-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2202.00666v1/

## Key points

- Language models underperform in text generation despite low perplexities on natural language corpora
- Understanding natural language as a communication channel can provide insights into probabilistic language generators
- Humans use language to efficiently communicate information while minimizing errors
- Probabilistic models should not always select words from the high-probability region, as it leads to dull and repetitive texts with low Shannon information content
- "Typical sampling" is introduced as a decision criterion for word selection during generation
- Typical sampling involves sampling from words with an information content close to its expected value, specifically close to the conditional entropy of the model
- Typical sampling aims to balance quality and diversity in generated text
- Automatic and human evaluations compared typical sampling with nucleus and top-k sampling methods
- Typical sampling offers competitive performance in terms of quality while reducing degenerate repetitions consistently
- Considering natural language as a communication channel is important for improving text generation by probabilistic models.

## Summary

In their paper titled "Typical Decoding for Natural Language Generation," authors Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell address the issue of language models underperforming in text generation despite achieving low perplexities on various natural language corpora. They propose that understanding natural language as a communication channel can provide insights into the behavior of probabilistic language generators. The authors argue that humans use language to efficiently communicate information while minimizing errors. Each word is chosen with the subconscious goal of conveying meaning effectively. They suggest that probabilistic models should mimic this behavior by not always selecting words from the high-probability region of the distribution, which often leads to dull or repetitive texts with low Shannon information content. To address this challenge, the authors introduce "typical sampling" as a decision criterion for word selection during generation. Typical sampling involves sampling from the set of words with an information content close to its expected value, specifically close to the conditional entropy of the model. This approach aims to balance quality and diversity in generated text. The researchers conducted automatic and human evaluations comparing typical sampling with nucleus and top-k sampling methods. The results showed that typical sampling offers competitive performance in terms of quality while consistently reducing degenerate repetitions. Overall, this work highlights the importance of considering natural language as a communication channel and proposes a novel approach, typical decoding, for improving text generation by probabilistic models.