# Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases

**Authors:** Aengus Lynch, Gbètondji J-S Dovonon, Jean Kaddour, Ricardo Silva  
**arXiv:** [2303.05470](https://arxiv.org/abs/2303.05470)  
**Date:** 2023-03-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.05470v3/

## Key points

- Spawrious is a benchmark dataset that addresses the problem of spurious correlations (SCs) in image classification.
- SCs occur when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data, leading to misclassifications during test time.
- The existing benchmark datasets suffer from various limitations, such as over-saturation or only containing one-to-one (O2O) SCs but no many-to-many (M2M) SCs arising between groups of spurious attributes and classes.
- To address these limitations, Spawrious presents \benchmark-\{O2O, M2M\}-\{Easy, Medium, Hard\}, an image classification benchmark suite containing spurious correlations between classes and backgrounds.
- The dataset contains approximately 152k high-quality images generated using a text-to-image model and an image captioning model to filter out unsuitable ones.
- The experimental results demonstrate that state-of-the-art group robustness methods struggle with Spawrious, particularly on the Hard splits where none of them achieved over 70% accuracy using a ResNet50 pretrained on ImageNet.
- By examining model misclassifications, researchers detected reliances on spurious backgrounds, demonstrating that Spawrious provides a significant challenge for image classification models.
- In addition to addressing background features' influence on SCs in image classification models, future work could instantiate desiderata with non-background spurious attributes and evaluate more generalization techniques on Spawrious.
- Overall, Spawrious provides a valuable benchmark dataset for evaluating the robustness of image classification models against spurious correlations arising from both one-to-one and many-to -many relationships between classes and background features.

## Summary

Spawrious is a benchmark dataset that addresses the problem of spurious correlations (SCs) in image classification. SCs occur when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data, leading to misclassifications during test time. The existing benchmark datasets suffer from various limitations, such as over-saturation or only containing one-to-one (O2O) SCs but no many-to-many (M2M) SCs arising between groups of spurious attributes and classes. To address these limitations, Spawrious presents \benchmark-\{O2O, M2M\}-\{Easy, Medium, Hard\}, an image classification benchmark suite containing spurious correlations between classes and backgrounds. The dataset contains approximately 152k high-quality images generated using a text-to-image model and an image captioning model to filter out unsuitable ones. The experimental results demonstrate that state-of-the-art group robustness methods struggle with Spawrious, particularly on the Hard splits where none of them achieved over 70% accuracy using a ResNet50 pretrained on ImageNet. By examining model misclassifications, researchers detected reliances on spurious backgrounds, demonstrating that Spawrious provides a significant challenge for image classification models. In addition to addressing background features' influence on SCs in image classification models, future work could instantiate desiderata with non-background spurious attributes and evaluate more generalization techniques on Spawrious. These techniques include different robustness penalties, meta learning unsupervised domain adaptation dropout flat minima weight averaging (counterfactual) data augmentation. Overall, Spawrious provides a valuable benchmark dataset for evaluating the robustness of image classification models against spurious correlations arising from both one-to-one and many-to -many relationships between classes and background features.