# Scalable Extraction of Training Data from (Production) Language Models

**Authors:** Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, Katherine Lee  
**arXiv:** [2311.17035](https://arxiv.org/abs/2311.17035)  
**Date:** 2023-11-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.17035v1/

## Key points

- Extractable memorization: ability to extract training data from machine learning models without prior knowledge of the dataset
- Adversaries can extract gigabytes of training data from various language models (Pythia, GPT-Neo, LLaMA, Falcon, ChatGPT)
- Existing techniques effective for attacking unaligned models
- New divergence attack strategy developed for attacking aligned models like ChatGPT
- Divergence attack causes model to deviate from chatbot-style responses and emit training data at a significantly higher rate (150 times more)
- Practical attacks can recover more data than previously anticipated
- Current alignment techniques do not eliminate memorization entirely
- Findings emphasize the importance of addressing extractable memorization for improved security and privacy.

## Summary

This paper titled "Scalable Extraction of Training Data from (Production) Language Models" explores the concept of extractable memorization, which refers to the ability of an adversary to efficiently extract training data from a machine learning model without any prior knowledge of the training dataset. The authors demonstrate that adversaries can successfully extract gigabytes of training data from various types of language models, including open-source models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT. The study reveals that existing techniques from the literature are effective in attacking unaligned models. However, to attack aligned models such as ChatGPT, the authors develop a new divergence attack strategy. This technique causes the model to deviate from its typical chatbot-style responses and instead emit training data at a significantly higher rate—150 times more than when it behaves correctly. By employing these methods, the researchers demonstrate that practical attacks can recover far more data than previously anticipated. The findings also indicate that current alignment techniques do not eliminate memorization entirely. The results presented in this paper provide insight into the vulnerabilities of language models and emphasize the importance of addressing extractable memorization for improved security and privacy.