# Robust Feature-Level Adversaries are Interpretability Tools

**Authors:** Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, Gabriel Kreiman  
**arXiv:** [2110.03605](https://arxiv.org/abs/2110.03605)  
**Date:** 2021-10-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2110.03605v6/

## Key points

- Adversarial attacks in computer vision traditionally focus on pixel-level perturbations
- Recent research explores manipulation of latent representations of image generators for "feature-level" adversarial perturbations
- Feature-level attacks provide useful classes of inputs for studying representations in models
- These adversaries are highly robust and versatile, capable of producing targeted, universal, disguised, physically-realizable and black-box attacks at the ImageNet scale
- Adversarial images can serve as a practical interpretability tool for identifying bugs in networks by making predictions about spurious associations between features and classes
- Copy/paste attacks involve inserting one natural image into another to cause unexpected misclassification and are useful for physically-realizable attacks
- Researchers selected a source and target class and generated class-universal adversarial features before manually analyzing them for motifs resembling natural objects to develop copy/paste attacks
- Comparisons with other methods show that this approach allows for targeted attacks and generates adversarial features conditional on any distribution over source images with which adversaries are trained.
- Feature-level adversarial attacks hold promise as an approach for rigorous interpretability research; supporting the design of tools to better understand what a model has learned and diagnose brittle feature associations.
- Code used in this study is available at https://github.com/thestephencasper/feature_level_adv

## Summary

Adversarial attacks in computer vision have traditionally focused on pixel-level perturbations, which can be difficult to interpret. However, recent research has explored the manipulation of latent representations of image generators to create "feature-level" adversarial perturbations that are more perceptible and interpretable. In this regard, a new study makes three significant contributions to the field. Firstly, it observes that feature-level attacks provide useful classes of inputs for studying representations in models. Secondly, it demonstrates that these adversaries are highly robust and versatile, capable of producing targeted, universal, disguised, physically-realizable and black-box attacks at the ImageNet scale. Thirdly, it shows how these adversarial images can serve as a practical interpretability tool for identifying bugs in networks by making predictions about spurious associations between features and classes. The study uses copy/paste attacks to validate interpretations made using feature-level adversarial attacks. Copy/paste attacks involve inserting one natural image into another to cause an unexpected misclassification. These types of attacks are restricted compared to patch attacks because the features pasted into an image must be natural objects. However, they are highly relevant for physically-realizable attacks because they suggest combinations of real objects that yield unexpected classifications. To develop copy/paste attacks in this study, researchers selected a source and target class and generated class-universal adversarial features before manually analyzing them for motifs resembling natural objects. The success of these copy/paste attacks demonstrated their usefulness for interpreting the target network since they required human understanding of the mistake the model was making well enough to manually exploit it. Comparisons with other methods show that while prior works have developed copy/paste attacks via interpretability tools like feature visualization-based methods inspired by [7], our approach allows for targeted attacks and generates adversarial features conditional on any distribution over source images with which adversaries are trained. Overall, this study suggests that feature-level adversarial attacks hold promise as an approach for rigorous interpretability research; supporting the design of tools to better understand what a model has learned and diagnose brittle feature associations. The code used in this study is available at https://github.com/thestephencasper/feature_level_adv .