# More than you've asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models

**Authors:** Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, Mario Fritz  
**arXiv:** [2302.12173](https://arxiv.org/abs/2302.12173)  
**Date:** 2023-02-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.12173v1/

## Key points

- Significant advancements in Large Language Models (LLMs) are being integrated into various systems such as IDEs and search engines.
- LLMs can be controlled through natural language prompts, but their internal functionality remains implicit and unassessable.
- Prompt Injection (PI) attacks have been introduced as a way to misalign LLMs, allowing adversaries to manipulate them and produce malicious content.
- Application-Integrated LLMs augment LLMs with retrieval and API calling capabilities, enabling them to process poisoned content retrieved from the web.
- Attackers can indirectly perform PI attacks using Application-Integrated LLMs by injecting pre-selected malicious prompts into the retrieved content.
- The authors systematically analyze the threat landscape of Application-Integrated LLMs and discuss new attack vectors to validate their practical viability.
- Synthetic applications are used to demonstrate specific scenarios within this research.
- There is an urgent need for evaluating current mitigation techniques and investigating whether new techniques are required to defend against these threats.

## Summary

We are witnessing significant advancements in Large Language Models (LLMs), which are being integrated into various systems such as integrated development environments (IDEs) and search engines. These LLMs can be controlled through natural language prompts, but their internal functionality remains implicit and unassessable. While this adaptability makes them suitable for unseen tasks, it also exposes them to targeted adversarial prompting. Prompt Injection (PI) attacks have recently been introduced as a way to misalign LLMs. In these attacks, an adversary can manipulate the LLM to produce malicious content or override the original instructions and filtering schemes. However, these attacks have assumed that the adversary directly prompts the LLM. This work introduces a new set of attack vectors by augmenting LLMs with retrieval and API calling capabilities, known as Application-Integrated LLMs. These LLMs can process poisoned content retrieved from the web, which contains pre-injected malicious prompts selected by adversaries. The authors demonstrate that attackers can indirectly perform PI attacks using these Application-Integrated LLMs. Based on this insight, the authors systematically analyze the resulting threat landscape of Application-Integrated LLMs and discuss various new attack vectors to validate their practical viability. To further evaluate their findings, they implement specific demonstrations within synthetic applications. In summary, this work highlights the urgent need for evaluating current mitigation techniques and investigating whether new techniques are required to defend against these threats to LLMs.