# Coercing LLMs to do and reveal (almost) anything

**Authors:** Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, Tom Goldstein  
**arXiv:** [2402.14020](https://arxiv.org/abs/2402.14020)  
**Date:** 2024-02-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.14020v1/

## Key points

- Authors: Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, Tom Goldstein
- Topic: Adversarial attacks on large language models (LLMs)
- Attacks go beyond "jailbreaking" to make harmful statements
- Overview of attack surfaces and goals for coercing LLMs
- Categorization of attacks leading to misdirection, model control, denial-of-service, data extraction
- Controlled experiments reveal many attacks stem from pre-training LLMs with coding capabilities
- Identification of security risks posed by "glitch" tokens in LLM vocabularies
- Importance of understanding and mitigating risks to prevent malicious manipulations
- Need for enhanced security measures in training and deploying LLMs

## Summary

In their paper titled "Coercing LLMs to do and reveal (almost) anything," authors Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein delve into the realm of adversarial attacks on large language models (LLMs). They highlight that these attacks go beyond merely "jailbreaking" the model to make harmful statements. The authors provide a comprehensive overview of the various attack surfaces and goals that can be targeted when coercing LLMs. Through a series of concrete examples, the paper categorizes and systematizes different types of attacks that can lead to unintended behaviors such as misdirection, model control, denial-of-service, or data extraction. The authors conduct controlled experiments to analyze these attacks and discover that many stem from the practice of pre-training LLMs with coding capabilities. Additionally, they point out the presence of peculiar "glitch" tokens in common LLM vocabularies that pose security risks and should be eliminated. The research sheds light on the broader spectrum of adversarial threats faced by LLMs and emphasizes the importance of understanding and mitigating these risks in order to safeguard against potential malicious manipulations. The findings underscore the need for enhanced security measures in training and deploying large language models to prevent coercion into undesirable actions or disclosures.