# Tokenization and the Noiseless Channel

**Authors:** Vilém Zouhar, Clara Meister, Juan Luis Gastaldi, Li Du, Mrinmaya Sachan, Ryan Cotterell  
**arXiv:** [2306.16842](https://arxiv.org/abs/2306.16842)  
**Date:** 2023-06-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.16842v1/

## Key points

- Importance of subword tokenization in NLP pipelines
- Lack of understanding regarding superior performance of certain tokenizers and hyperparameters
- Effective tokenizers contribute to efficient channel usage in transmitting input to the model
- Quantification of efficiency using information-theoretic measures like Shannon entropy ratio
- Optimal encoding based on Shannon entropy assigns lengthy codes to low-frequency tokens and short codes to high-frequency tokens
- Introduction of refined efficiency concept utilizing Rényi entropy to penalize extreme frequency distributions
- Strong correlation between Rényi entropy with α = 2.5 and BLEU scores (0.78) in machine translation tasks
- Replication of findings using metrics like CHRF, BLEURT, and COMET in additional experiments detailed in Table 2 of the paper's appendix
- Compression principle advocating for alignment of downstream task metrics with expected code lengths while penalizing long codewords for infrequent tokens
- Provision of a user-friendly package for evaluating tokenizations and insights into how different entropy measures impact model performance in NLP tasks

## Summary

In the study "Tokenization and the Noiseless Channel," researchers Vilém Zouhar, Clara Meister, Juan Luis Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell delve into the importance of subword tokenization in NLP pipelines. They highlight the lack of understanding regarding why certain combinations of tokenizers and hyperparameters result in superior performance in downstream models compared to others. The researchers propose that effective tokenizers contribute to efficient channel usage, where the channel serves as the medium through which input is transmitted to the model. Efficiency is quantified using information-theoretic measures such as the ratio of Shannon entropy to the maximum possible entropy of the token distribution. The study explores how an optimal encoding based on Shannon entropy tends to assign lengthy codes to low-frequency tokens and short codes to high-frequency tokens. To address this issue, the researchers introduce a refined concept of efficiency utilizing Rényi entropy, which penalizes distributions containing extremely high or low-frequency tokens. In their experiments focusing on machine translation tasks, they discover a strong correlation between Rényi entropy with α = 2.5 and BLEU scores (0.78), outperforming correlations with compressed length (-0.32). Furthermore, the researchers replicate their findings using metrics like CHRF, BLEURT, and COMET in additional experiments detailed in Table 2 of their paper's appendix. They introduce the compression principle advocating for downstream task metrics like BLEU to align with expected code lengths while penalizing long codewords associated with infrequent tokens. In conclusion, they provide a user-friendly package for evaluating tokenizations and offer insights into how different entropy measures can impact model performance in NLP tasks like machine translation from German to English using large parallel datasets from CommonCrawl. Their work underscores the significance of considering channel efficiency and entropy measures when designing tokenization strategies for NLP applications.