# Stealing Part of a Production Language Model

**Authors:** Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, Florian Tramèr  
**arXiv:** [2403.06634](https://arxiv.org/abs/2403.06634)  
**Date:** 2024-03-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.06634v1/

## Key points

- Groundbreaking model-stealing attack can extract precise and nontrivial information from black-box production language models
- Targets the embedding projection layer of transformer models, including popular models like OpenAI's ChatGPT and Google's PaLM-2
- Successfully extracted entire projection matrix of OpenAI's Ada and Babbage language models for less than $20 USD, revealing hidden dimensions of 1024 and 2048 respectively
- Uncovered exact hidden dimension size of gpt-3.5-turbo model through attack methodology, estimating cost under $2,000 in queries to recover entire projection matrix
- Potential vulnerabilities in widely used language models exposed
- Paper discusses potential defenses and mitigations against such attacks, as well as future work to enhance or extend attack methodology
- Collaborative effort of multiple authors underscores significance of research in uncovering security risks associated with black-box language models

## Summary

The paper "Stealing Part of a Production Language Model" presents a groundbreaking model-stealing attack that can extract precise and nontrivial information from black-box production language models. This includes popular models such as OpenAI's ChatGPT and Google's PaLM-2. The attack specifically targets the embedding projection layer of transformer models, even with only typical API access available. Remarkably, the researchers were able to successfully extract the entire projection matrix of OpenAI's Ada and Babbage language models for less than \$20 USD. This revealed hidden dimensions of 1024 and 2048 respectively, shedding light on previously unknown details about their internal structures. Additionally, the study also uncovered the exact hidden dimension size of the gpt-3.5-turbo model through their attack methodology. By estimating that it would cost under \$2,000 in queries to recover the entire projection matrix of this model, potential vulnerabilities in widely used language models are exposed. The paper concludes by discussing potential defenses and mitigations against such attacks and explores future work that could further enhance or extend their attack methodology. The collaborative effort of authors including Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick and Florian Tramèr underscores the significance of this research in uncovering security risks associated with black-box language models.