# Mirostat: A Perplexity-Controlled Neural Text Decoding Algorithm

**Authors:** Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, Lav R. Varshney  
**arXiv:** [2007.14966](https://arxiv.org/abs/2007.14966)  
**Date:** 2020-07-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2007.14966v1/

## Key points

- Neural text decoding is important for generating high-quality texts using language models
- Popular decoding algorithms like top-k, top-p (nucleus), and temperature-based sampling can truncate or distort the unreliable low probability tail of the language model
- The authors provide a theoretical analysis of perplexity in top-k, top-p, and temperature sampling
- They propose a feedback-based adaptive top-k text decoding algorithm called mirostat that allows for generating text with any length while maintaining a predetermined value of perplexity
- Mirostat enables the generation of high-quality text without extensive parameter tuning
- Experiments show that perplexity drops significantly with generated text length for low values of k and p in top-k and top-p sampling, but mirostat successfully avoids this issue
- Cross-entropy has a near-linear relation with repetition in generated text across different sampling methods, although it slightly depends on the specific language model used
- Mirostat offers control over repetitions and helps generate high-quality texts without arbitrary parameter tuning

## Summary

Neural text decoding plays a crucial role in generating high-quality texts using language models. However, popular decoding algorithms like top-k, top-p (nucleus), and temperature-based sampling often truncate or distort the unreliable low probability tail of the language model. While these methods can generate high-quality text after parameter tuning, they are ad hoc and lack control over the statistics of the output. To address this issue, the authors of the paper provide a theoretical analysis of perplexity in top-k, top-p, and temperature sampling. They find that cross-entropy behaves approximately linearly as a function of p in top-p sampling, while it is a nonlinear function of k in top-k sampling under Zipfian statistics. Based on this analysis, they propose a feedback-based adaptive top-k text decoding algorithm called mirostat. This algorithm allows for generating text with any length while maintaining a predetermined value of perplexity. By doing so, it enables the generation of high-quality text without requiring extensive parameter tuning. The experiments conducted by the authors reveal interesting insights. For low values of k and p in top-k and top-p sampling, perplexity drops significantly with generated text length. However, mirostat successfully avoids both traps. The experiments demonstrate that cross-entropy has a near-linear relation with repetition in generated text. This relationship remains consistent across different sampling methods but slightly depends on the specific language model used. In conclusion,, mirostat offers control over repetitions and helps generate high-quality texts without relying on arbitrary parameter tuning. This research contributes valuable insights into understanding and improving the decoding process of language models.