The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy

   Tong Wu, Shu-Jian Zhang, Kaiqiang Song et al. · 2024 · International Conference on Learning Representations · cited 91 times · Not summarized yet

   Abstract

   Large Language Models (LLMs) are susceptible to security and safety threats, such as prompt injection, prompt extraction, and harmful requests. One major cause of these vulnerabilities is the lack of an instruction hierarchy. Modern LLM architectures treat all inputs equally, failing to distinguish between and prioritize various types of instructions, such as system messages, user prompts, and data. As a result, lower-priority user prompts may override more critical system instructions, including safety protocols. Existing approaches to achieving instruction hierarchy, such as delimiters and instruction-based training, do not address this issue at the architectural level. We introduce the In…

2. Very close Cites this paper Uses this paper in depth Not relevant

   Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization

   L. Bolliger, L. Jäger · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   Production LLMs receive instructions from sources with very different levels of trust, yet attend to every token with uniform architectural privilege. This is the structural vulnerability that enables malicious prompt injections and, more broadly, leaves models without a principled way to resolve conflicts between legitimate but competing instructions. A common training-based response is to teach models an explicit instruction hierarchy; existing approaches, however, formalize hierarchies of only three or four levels, treat all violations as equally severe, and rarely evaluate the full set of pairwise level interactions. We formalize a k-level instruction hierarchy problem and instantiate it…

3. Close Not relevant

   Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning

   Zhi-Yuan Chang, Ming-Yang Li, Yue-Kai Huang et al. · 2026 · Annual Meeting of the Association for Computational Linguistics · cited 1 time · Not summarized yet

   Abstract

   Large language model (LLM)-integrated applications have become increasingly prevalent, yet face critical security vulnerabilities from prompt injection (PI) attacks. Defending against PI attacks faces two major issues: malicious instructions can be injected through diverse vectors, and injected instructions often lack clear semantic boundaries from the surrounding context, making them difficult to identify. To address these issues, we propose InstruCoT, a model enhancement method for PI defense that synthesizes diverse training data and employs instruction-level chain-of-thought fine-tuning, enabling LLMs to effectively identify and reject malicious instructions regardless of their source or…

4. Close Cites this paper Not relevant

   Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance

   Rui Wen, Jia-Yang Liu, Ze-Yu Yang et al. · 2026 · Not summarized yet

   Abstract

   When a prompt injection attack succeeds, a Large Language Model (LLM) abandons its assigned system role to comply with an adversarial instruction. While prior work has extensively quantified how often this occurs, we ask a more fundamental question: where inside the network does the model actually decide to break the rules? Using layer-by-layer causal activation patching across five models (4B to 32B parameters), we find a clear dissociation: attack information is linearly decodable from the first layer, yet causal leverage over the model's behavior is negligible until a late-layer bottleneck in the final third of the network. Patching this bottleneck reverses compliance in 77--92\% of cases…

5. Close Cited by this paper Not relevant

   Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

   Kai Greshake, Sahar Abdelnabi, Shailesh Mishra et al. · 2023 · Cryptography and Security · cited 2,408 times · AI summary available

   Abstract

   Large Language Models (LLMs) are increasingly being integrated into various applications. The functionalities of recent LLMs can be flexibly modulated via natural language prompts. This renders them susceptible to targeted adversarial prompting, e.g., Prompt Injection (PI) attacks enable attackers to override original instructions and employed controls. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We argue that LLM-Integrated Applications blur the line between data and instructions. We reveal new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (without a direct interface) exploit LLM-integr…

6. Close Cites this paper Uses this paper in depth Not relevant

   When Context Gets Root: Privilege Escalation in LLM Harnesses

   Xing-Bang He, Yuanwei Chen, Yi Qian et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Instruction hierarchy is a model-side defense that assigns instructions different levels of privilege according to their sources. These levels constrain which content may direct model behavior. During agent execution, however, agent harnesses construct context for each model invocation. This construction can elevate low-level content to a higher instruction level and grant it greater model-facing privilege. We introduce instruction privilege escalation. In this attack, an attacker induces an agent to elevate low-level malicious content to a higher instruction level. The elevated content then causes the agent to execute instructions it would not follow at their original level. We evaluate thi…

7. Close Cites this paper Not relevant

   How Language Models Choose Sides: Internal Representations of Instruction Hierarchy

   Enrique Balp-Straffon, Chih-Hao Hsu, Rushiraj Gadhvi et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Abstract

   We study how instruction-tuned LLMs arbitrate direct conflicts between system and user instructions. We introduce a benchmark of 41 paired constraints with deterministic verifiers and evaluate eight models under matched baseline, conflict, and same-channel control conditions. Behaviourally, the models split into three regimes by System Authority Delta: hierarchy-respecting models use the system channel as an authority signal, anti-hierarchy models follow the system less often than their same-channel baseline predicts, and no-effect models show little channel sensitivity. Llama-3.1-8B is the strongest anti-hierarchy case in our suite, following the system in only 0.10 of conflict trials. We u…

8. Close Cites this paper Uses this paper in depth Not relevant

   Improving Instruction Hierarchy by Learning from Goal-Goal Conflicts

   Si-Zhe Chen, Sanjay Kariyappa, Li-Wei Jiang et al. · 2026 · Not summarized yet

   Abstract

   The recent incident in which OpenAI agents autonomously compromised Hugging Face's infrastructure illustrates the importance of instruction hierarchy (IH): an LLM should follow lower-priority instructions, such as user requests, only when they do not conflict with higher-priority instructions, such as system policies. IH can be effectively improved by fine-tuning LLMs on simulated examples of instruction conflicts, whose diversity is key to teaching LLMs to properly handle conflicts at test time. However, our decomposition of a task into goals and constraints reveals a conflict type -- goal-goal conflicts -- that is absent from existing IH training sets from OpenAI and NVIDIA. We show the im…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game

   S. Toyer, Olivia Watkins, Ethan Mendes et al. · 2023 · International Conference on Learning Representations · cited 197 times · Not summarized yet

   Abstract

   While Large Language Models (LLMs) are increasingly being used in real-world applications, they remain vulnerable to prompt injection attacks: malicious third party prompts that subvert the intent of the system designer. To help researchers study this problem, we present a dataset of over 126,000 prompt injection attacks and 46,000 prompt-based"defenses"against prompt injection, all created by players of an online game called Tensor Trust. To the best of our knowledge, this is currently the largest dataset of human-generated adversarial examples for instruction-following LLMs. The attacks in our dataset have a lot of easily interpretable stucture, and shed light on the weaknesses of LLMs. We…

2. Close Used in depth by this paper Not relevant

   StruQ: Defending Against Prompt Injection with Structured Queries

   Si-Zhe Chen, Julien Piet, Chawin Sitawarin et al. · 2024 · USENIX Security Symposium · cited 545 times · Not summarized yet

   Abstract

   Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model into deviating from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that…

3. Close Used in depth by this paper Not relevant

   Prompt Injection attack against LLM-integrated Applications

   Yi Liu, Ge-Lei Deng, Yuekang Li et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 958 times · Not summarized yet

   Abstract

   Large Language Models (LLMs), renowned for their superior proficiency in language comprehension and generation, stimulate a vibrant ecosystem of applications around them. However, their extensive assimilation into various services introduces significant security risks. This study deconstructs the complexities and implications of prompt injection attacks on actual LLM-integrated applications. Initially, we conduct an exploratory analysis on ten commercial applications, highlighting the constraints of current attack strategies in practice. Prompted by these limitations, we subsequently formulate HouYi, a novel black-box prompt injection attack technique, which draws inspiration from traditiona…

4. Close Not relevant

   Ignore Previous Prompt: Attack Techniques For Language Models

   Fábio Perez, I. Ribeiro · 2022 · arXiv.org · preprint, not peer reviewed · cited 1,216 times · Not summarized yet

   Abstract

   Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's s…

5. Close Not relevant

   Jailbroken: How Does LLM Safety Training Fail?

   Alexander Wei, Nika Haghtalab, Jacob Steinhardt · 2023 · Machine Learning · cited 2,344 times · AI summary available

   Abstract

   Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evalua…

Papers that cite it

1. Close Uses this paper in depth Not relevant

   Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

   Ze-Sheng Shi, Xue Li, Chen-Rui Dai et al. · 2026 · cited 1 time · Not summarized yet

   Abstract

   LLMs over external data face a per-instruction tension: a single document or tool return often interleaves legitimate directives with adversarial prompt injections, and the model must follow the former while rejecting the latter. Existing defenses suppress or trust all external instructions, collapsing the safety-utility frontier and forcing a choice between over-refusal and over-compliance. We argue the missing capability is \textbf{per-instruction credit attribution} and formalize it as External-Data Instruction Classification, an identify-classify-act decision over each embedded directive. We propose \textbf{EDIC}, a decomposed reward framework for per-instruction compliance. A scene-grou…

2. Close Uses this paper in depth Not relevant

   SecPO: Principled Adversarial Training for Prompt Injection Security

   Si-Zhe Chen, Yi-Bo Peng, Jaewon Chang et al. · 2026 · cited 1 time · Not summarized yet

   Abstract

   Recognized as a top threat to AI agents, prompt injection allows instructions from untrusted data (email, documents, websites) to manipulate the agent's operation. The existing best-defended LLM, Meta-SecAlign, is trained with the DPO loss but still suffers from attack success rates (ASRs) approaching 100% under adaptive prompt injections. One reason is that DPO anchors the model around the undefended reference model, which is insecure. We show an idealized reference model---an injection-invariant, utility-preserving oracle---can nevertheless be approximated, as its output distribution in training is interestingly accessible. This new observation motivates a theoretically principled SecPO lo…

3. Close Uses this paper in depth Not relevant

   IHDec: Divergence-Steered Contrastive Decoding for Securing Multi-Turn Instruction Hierarchies

   Nicole Geumheon Liu, Haeun Jang, Yonghyun Jun et al. · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   Large Language Models (LLMs) often fail to maintain instruction hierarchies (IH) when processing multi-source inputs with varying role-level priorities, paradoxically adhering to lower-priority directives during conflicts. While existing defenses mitigate this issue, they are largely restricted to single-turn scenarios and require expensive fine-tuning. In this paper, we formalize this failure mode in multi-turn contexts via a Jensen-Shannon Divergence (JSD) framework, uncovering a pervasive role-influence inversion phenomenon where subordinate inputs override superior roles. To rectify this without training, we propose IHDec (Instruction Hierarchy-steered Decoding). IHDec leverages JSD to a…

4. Close Uses this paper in depth Not relevant

   IH-Benchmark: A Conflict-Centered Benchmark for Instruction-Hierarchy Robustness in LLM Applications

   Conor McCauley, Zeliang Kan, Jason Martin · 2026 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet

   Abstract

   When a language model receives conflicting instructions from different priority levels, which one does it actually follow? This question lies at the heart of reliable LLM deployment. Existing benchmarks answer this only partially, often focusing on a single hierarchy edge or adapting public datasets with limited tool-use coverage. We present IH-Benchmark, a conflict-centered benchmark for instruction-hierarchy robustness across direct system-user conflicts (S>U) and tool-mediated user-tool (U>T) conflicts. IH-Benchmark is built from a human-authored taxonomy of 44 constraint families across generic, health, finance, retail, and coding settings, and evaluates scenarios with a uniform binary p…

5. Close Not relevant

   Combating Instruction Conflict via Energy-Driven Latent Conflict Detection

   Mi Ma, Yu-Xin Wu, Jing-Bo Wang et al. · 2026 · Not summarized yet

   Abstract

   Large Language Models (LLMs) are increasingly deployed with hierarchical instructions, yet they remain vulnerable to conflicts in which user directives override system-level constraints. Existing defense mechanisms predominantly focus on static input inspection and therefore fail to detect Response Drift, a phenomenon in which the model's final response violates system-level constraints despite seemingly compliant inputs. To bridge this gap, we introduce ELCD, a response-level latent conflict detector for post-generation, pre-delivery verification. Given the full generated output, ELCD constructs a composite hidden-state representation by concatenating the final-token embedding with the mean…