# Deduplicating Training Data Makes Language Models Better

**Authors:** Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini  
**arXiv:** [2107.06499](https://arxiv.org/abs/2107.06499)  
**Date:** 2021-07-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2107.06499v1/

## Key points

- Language models trained on existing datasets have over 1% of their output directly copied from the training data
- Authors develop two tools to deduplicate training datasets
- A single 61-word English sentence repeated more than 60,000 times is successfully removed from the C4 dataset using deduplication techniques
- Deduplication reduces memorized text production by language models ten times less frequently while maintaining or improving accuracy
- Deduplication also reduces train-test overlap in standard datasets by more than 4%, leading to more accurate evaluation of language models
- Code for reproducing the study and performing dataset deduplication is available on GitHub (https://github.com/google-research/deduplicate-text-datasets)
- This study emphasizes the importance of addressing duplicate examples and repetitive substrings in language modeling datasets for improved performance and reliable evaluations in natural language processing tasks.

## Summary

In their paper titled "Deduplicating Training Data Makes Language Models Better," authors Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch and Nicholas Carlini address the issue of near-duplicate examples and long repetitive substrings in existing language modeling datasets. They find that over 1% of the unprompted output generated by language models trained on these datasets is directly copied from the training data. To tackle this problem, the authors develop two tools that enable them to deduplicate training datasets. For instance, they successfully remove a single 61-word English sentence that is repeated more than 60,000 times from the C4 dataset. By implementing deduplication techniques, they are able to train models that produce memorized text ten times less frequently while achieving the same or even better accuracy. Additionally, deduplication helps reduce train-test overlap in standard datasets by more than 4%, leading to more accurate evaluation of language models. The authors provide code for reproducing their work and performing dataset deduplication on GitHub (https://github.com/google-research/deduplicate-text-datasets). This code release allows other researchers and practitioners to benefit from their findings and improve the quality of language models. Overall, this study highlights the importance of addressing duplicate examples and repetitive substrings in language modeling datasets for improved performance and reliable evaluations in natural language processing tasks.