# Model Dementia: Generated Data Makes Models Forget

**Authors:** Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson  
**arXiv:** [2305.17493](https://arxiv.org/abs/2305.17493)  
**Date:** 2023-05-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.17493v1/

## Key points

- Large language models (LLMs) like GPT-2, GPT-3(.5), and GPT-4 have revolutionized image creation from descriptive text.
- LLMs have demonstrated remarkable performance across various language tasks and have been introduced to the general public through platforms like ChatGPT.
- The concept of "model dementia" refers to irreversible defects in generative models caused by using model-generated content in training.
- Model dementia can occur not only in LLMs but also in Variational Autoencoders (VAEs) and Gaussian Mixture Models (GMMs).
- Learning with generated data is possible, but it leads to degraded performance in models over generations.
- Just training a single moderately large model produces twice the American lifetime worth of CO2; hence an experiment demonstrating model dementia was not run.
- Data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.
- It is essential to take model dementia seriously if we are to sustain the benefits of training from large-scale data scraped from the web.

## Summary

The use of large language models (LLMs) such as GPT-2, GPT-3(.5), and GPT-4 has revolutionized the creation of images from descriptive text. These models have demonstrated remarkable performance across various language tasks and have been introduced to the general public through platforms like ChatGPT. However, as LLMs contribute more to the language found online, it is crucial to consider their impact on the whole ecosystem of online text and images. In a recent paper, researchers explore the concept of "model dementia," which refers to irreversible defects in generative models caused by using model-generated content in training. The study shows that this effect can occur in not only LLMs but also Variational Autoencoders (VAEs) and Gaussian Mixture Models (GMMs). The authors build theoretical intuition behind this phenomenon and demonstrate its ubiquity amongst all learned generative models. The study finds that learning with generated data is possible, but it leads to degraded performance in models over generations. Models tend to produce samples that would be produced with higher probabilities by the original model while introducing their own improbable sequences or errors. The research team notes that just training a single moderately large model produces twice the American lifetime worth of CO2; hence they opted not to run an experiment demonstrating model dementia. The study highlights that data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet. Therefore, it is essential to take model dementia seriously if we are to sustain the benefits of training from large-scale data scraped from the web. In conclusion, while LLMs have transformed image creation from descriptive text and demonstrated astonishing performance across various language tasks, their impact on generative models must be considered carefully. The concept of "model dementia" highlights irreversible defects caused by using model-generated content in training. As such, collecting genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.