# SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents

**Authors:** Ethan Rathbun, Christopher Amato, Alina Oprea  
**arXiv:** [2405.20539](https://arxiv.org/abs/2405.20539)  
**Date:** 2024-05-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.20539v2/

## Key points

- Reinforcement learning (RL) is increasingly used in real-world, safety-critical scenarios
- Focus on backdoor poisoning attacks in RL agents during training to manipulate behavior at inference time
- Theoretical limitations of existing work in generalizing across domains and Markov Decision Processes (MDPs)
- Introduction of a novel poisoning attack framework aligning adversary's objectives with finding an optimal policy for long-term success
- Introduction of "SleeperNets" as a universal backdoor attack strategy using dynamic reward poisoning techniques
- Evaluation of SleeperNets attack in various environments showing improved success rates while maintaining benign episodic return
- Formal analysis of static reward poisoning attacks' weaknesses
- Introduction of an "outer-loop" threat model for more informed poisoning attacks after each episode
- Development of a novel framework utilizing dynamic reward poisoning for creating RL backdoor attacks with provable guarantees of success and stealth over time

## Summary

, , , , 

Reinforcement learning (RL) is a rapidly evolving field with increasing applications in real-world, safety-critical scenarios. As the use of RL algorithms grows, it becomes crucial to ensure their robustness against adversarial attacks. In this study, the focus is on backdoor poisoning attacks, a stealthy form of training-time attacks against RL agents. These attacks involve adversaries intervening during the training process to manipulate the agent's behavior so that it reliably performs a specific action when presented with a predetermined trigger at inference time. The research uncovers theoretical limitations in existing work by demonstrating their inability to generalize across different domains and Markov Decision Processes (MDPs). Motivated by this discovery, a novel poisoning attack framework is formulated. This framework aligns the adversary's objectives with finding an optimal policy, ensuring attack success in the long run. Leveraging insights from theoretical analysis, "SleeperNets" is introduced as a universal backdoor attack strategy. SleeperNets exploits a newly proposed threat model and employs dynamic reward poisoning techniques to achieve its goals. The study evaluates the SleeperNets attack in various environments spanning multiple domains such as robotic navigation, video game playing, self-driving tasks, and stock trading. Results demonstrate significant improvements in attack success rates compared to existing methods while maintaining benign episodic return. The research also includes
1. The first formal analysis of static reward poisoning attacks, highlighting their weaknesses. 2. Introduction of an "outer-loop" threat model where adversaries manipulate agent rewards and state observations after each episode for more informed poisoning attacks. 3. Development of a novel framework utilizing dynamic reward poisoning for creating RL backdoor attacks with provable guarantees of success and stealth over time. Overall, the study provides valuable insights into enhancing the security of RL algorithms against adversarial threats through innovative backdoor poisoning attack strategies like SleeperNets.