# Formalizing and Benchmarking Prompt Injection Attacks and Defenses

**Authors:** Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, Neil Zhenqiang Gong  
**arXiv:** [2310.12815](https://arxiv.org/abs/2310.12815)  
**Date:** 2023-10-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.12815v3/

## Key points

- Prompt injection attacks are a type of cyber attack that aims to insert harmful instructions or data into the input of an LLM-Integrated Application.
- Previous studies on prompt injection attacks have primarily focused on case studies and lacked a formal framework for understanding these attacks and their defenses.
- A new study proposes a comprehensive framework for formalizing prompt injection attacks, encompassing existing attack strategies and allowing for the design of new attacks by combining different techniques.
- The study conducted a systematic evaluation of five prompt injection attacks using ten different LLMs across seven tasks, with a newly designed attack consistently outperforming existing attacks in various target tasks and injected data scenarios.
- Ten defense mechanisms against prompt injection attacks were systematically benchmarked, revealing that existing defenses are inadequate in effectively preventing or detecting these attacks.

## Summary

, , , , 

Prompt injection attacks, also known as malicious injections, are a type of cyber attack that aims to insert harmful instructions or data into the input of an LLM-Integrated Application. This can manipulate the results produced by the application and potentially compromise its functionality. While there have been previous studies on prompt injection attacks, they have primarily focused on case studies and lacked a formal framework for understanding these attacks and their defenses. To address this gap, a new study proposes a comprehensive framework for formalizing prompt injection attacks. This framework not only encompasses existing attack strategies but also allows for the design of new attacks by combining different techniques. The study then conducts a systematic evaluation of five prompt injection attacks using ten different LLMs across seven tasks. One notable finding is that a newly designed attack inspired by the proposed framework consistently outperforms existing attacks in various target tasks and injected data scenarios. This highlights the effectiveness of utilizing a structured framework for designing and evaluating prompt injection attacks. In addition to evaluating attacks, the study also systematically benchmarks ten defense mechanisms against prompt injection attacks. These defenses include both prevention-based strategies, which aim to redesign task prompts or preprocess data to thwart injected instructions, and detection-based methods, which focus on identifying compromised data within tasks. However, the evaluation reveals that existing defenses are inadequate in effectively preventing or detecting prompt injection attacks. This indicates a need for more robust defense mechanisms in this domain. Overall, this study makes significant contributions by introducing a formalized framework for prompt injection attacks, conducting thorough evaluations of both attacks and defenses using diverse LLMs and tasks, and providing an open-source platform for further research in this area. By establishing a common benchmark for evaluating future prompt injection attacks and defenses, this work sets the stage for advancements in cybersecurity measures against malicious injections in LLM-Integrated Applications.