# Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

**Authors:** Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, Sanmi Koyejo  
**arXiv:** [2404.01413](https://arxiv.org/abs/2404.01413)  
**Date:** 2024-04-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.01413v1/

## Key points

- Pretraining generative models on vast amounts of web-scale data can lead to model collapse when the models are trained on their own generated outputs.
- Model collapse is a phenomenon where a model's performance deteriorates with each iteration of fitting until it becomes essentially useless.
- Accumulating data serves as a safeguard against model collapse, preventing the linear increase in test error that occurs when data is replaced over time.
- Empirical testing across various model sizes, architectures, and hyperparameters shows that accumulating data effectively prevents model collapse in deep generative models.
- Accumulating data was found to mitigate model collapse in language models pretrained on text corpora, diffusion models for molecule generation, and variational autoencoders for image generation.
- The study by Matthias Gerstgrasser and colleagues provides theoretical insights and empirical evidence supporting the effectiveness of accumulating data to combat model collapse in generative modeling tasks.

## Summary

In the realm of generative models, the practice of pretraining on vast amounts of web-scale data has raised a critical question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops have unveiled a concerning phenomenon known as model collapse. This occurs when the performance of a model progressively deteriorates with each iteration of fitting until it becomes essentially useless. However, prior studies on model collapse have typically assumed that new data replaces old data over time, rather than considering the accumulation of data over time. In a recent paper titled "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data," researchers delve into this issue by comparing two scenarios: one where data is replaced and another where data accumulates. The study reveals that accumulating data serves as a safeguard against model collapse. By examining an analytically tractable setup involving a sequence of linear models fitted to previous models' predictions, the researchers demonstrate that while replacing data leads to a linear increase in test error with each iteration, accumulating data ensures that the test error remains bounded regardless of the number of iterations. Empirical testing further corroborates these findings, showing that accumulating data effectively prevents model collapse across various model sizes, architectures, and hyperparameters. The researchers conducted experiments using language models pretrained on text corpora and found that replacing data indeed triggers model collapse, whereas accumulating data successfully mitigates this issue. Moreover, similar results were observed in other deep generative models applied to real-world datasets, such as diffusion models for molecule generation and variational autoencoders for image generation. This comprehensive study by Matthias Gerstgrasser and colleagues provides both theoretical insights and empirical evidence supporting the notion that accumulating data can effectively combat model collapse in generative modeling tasks. By shedding light on this crucial aspect of training deep learning models, the research contributes valuable knowledge to the field and offers practical strategies for enhancing model robustness and performance in complex machine learning applications.