# SneakyPrompt: Evaluating Robustness of Text-to-image Generative Models' Safety Filters

**Authors:** Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, Yinzhi Cao  
**arXiv:** [2305.12082](https://arxiv.org/abs/2305.12082)  
**Date:** 2023-05-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.12082v1/

## Key points

- Authors Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao focus on text-to-image generative models and the challenges of filtering out Not-Safe-for-Work (NSFW) content.
- Models like Stable Diffusion and DALL$\cdot$E 2 have practical applications but struggle with NSFW content filtering.
- Researchers commonly use safety filters based on textual or visual cues to block NSFW content.
- SneakyPrompt is introduced as an automated attack framework to assess the robustness of safety filters in cutting-edge text-to-image generative models.
- SneakyPrompt utilizes reinforcement learning to craft prompts that generate NSFW images capable of evading existing safety filters.
- Evaluation shows SneakyPrompt's effectiveness in generating NSFW content using DALL$\cdot$E 2 with its default closed-box safety filter enabled.
- SneakyPrompt surpasses existing adversarial attacks in query efficiency and image quality when tested against open-source state-of-the-art safety filters on a Stable Diffusion model.
- The study highlights the importance of robust security measures in AI systems to address vulnerabilities in safety filters.

## Summary

In their paper titled "SneakyPrompt: Evaluating Robustness of Text-to-image Generative Models' Safety Filters," authors Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao delve into the realm of text-to-image generative models and the challenges posed by Not-Safe-for-Work (NSFW) content generation. The advent of models like Stable Diffusion and DALL$\cdot$E 2 has sparked significant interest due to their practical applications in various real-world scenarios. However, one major hurdle faced by these models is the need to filter out NSFW content such as violence and adult themes. To address this issue, researchers commonly deploy safety filters that block NSFW content based on either textual or visual cues. Previous studies have explored ways to bypass these safety filters; however, most approaches have been manual and tailored specifically to Stable Diffusion's official safety filter. The authors note that the bypass ratio of Stable Diffusion's safety filter stands at a mere 23.51% based on their evaluations. In response to this challenge, the authors introduce SneakyPrompt as an automated attack framework designed to assess the robustness of real-world safety filters in cutting-edge text-to-image generative models. The key innovation behind SneakyPrompt lies in its ability to identify alternative tokens within a prompt that lead to the generation of NSFW images capable of evading existing safety filters. Leveraging reinforcement learning (RL), SneakyPrompt guides an agent towards crafting prompts with positive rewards based on semantic similarity and successful bypasses. The evaluation conducted by the authors demonstrates SneakyPrompt's efficacy in generating NSFW content using an online model like DALL$\cdot$E 2 with its default closed-box safety filter enabled. Additionally, several open-source state-of-the-art safety filters were deployed on a Stable Diffusion model, revealing that SneakyPrompt not only successfully generates NSFW content but also surpasses existing adversarial attacks in terms of query efficiency and image quality. Overall, "SneakyPrompt" presents a novel approach towards evaluating the resilience of safety filters in text-to-image generative models against NSFW content generation. This study sheds light on potential vulnerabilities and highlights the importance of robust security measures in AI systems.