# MEMO: Test Time Robustness via Adaptation and Augmentation

**Authors:** Marvin Zhang, Sergey Levine, Chelsea Finn  
**arXiv:** [2110.09506](https://arxiv.org/abs/2110.09506)  
**Date:** 2021-10-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2110.09506v1/

## Key points

- Neural networks have high accuracy on in-distribution test points in deep learning.
- Real-world applications require models to be robust even when faced with unexpected perturbations or distribution shifts in the input.
- Test time robustification is a technique that uses the test input to improve model robustness.
- Prior works have proposed methods for test time adaptation, but they often introduce additional assumptions that limit their widespread adoption.
- Authors Marvin Zhang, Sergey Levine, and Chelsea Finn aim to devise methods that make no assumptions about the model training process and are broadly applicable at test time.
- The authors propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable.
- Different data augmentations are performed on the data point, and then all of the model parameters are adapted by minimizing the entropy of the model's average output distribution across these augmentations.
- Experimental results demonstrate that this approach consistently improves robust ResNet and vision transformer models by achieving accuracy gains of 1-8% over standard model evaluation.
- The proposed method also generally outperforms prior augmentation and adaptation strategies for various types of image corruptions (ImageNet-C), renditions of common objects (ImageNet-R), and adversarially chosen natural examples (ImageNet-A).
- This approach can help improve model performance under distribution shift scenarios without requiring additional assumptions about training processes or access to multiple test points.

## Summary

In the field of deep learning, neural networks have shown impressive accuracy on in-distribution test points. However, many real-world applications require models to be robust even when faced with unexpected perturbations or distribution shifts in the input. Test time robustification is a technique that uses the test input to improve model robustness. While recent prior works have proposed methods for test time adaptation, they often introduce additional assumptions that limit their widespread adoption. In this study, authors Marvin Zhang, Sergey Levine, and Chelsea Finn aim to devise methods that make no assumptions about the model training process and are broadly applicable at test time. The authors propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable. When presented with a test example, different data augmentations are performed on the data point, and then all of the model parameters are adapted by minimizing the entropy of the model's average output distribution across these augmentations. This objective encourages the model to make consistent predictions across different augmentations, enforcing invariances encoded in these augmentations while maintaining confidence in its predictions. Experimental results demonstrate that this approach consistently improves robust ResNet and vision transformer models by achieving accuracy gains of 1-8% over standard model evaluation. The proposed method also generally outperforms prior augmentation and adaptation strategies for various types of image corruptions (ImageNet-C), renditions of common objects (ImageNet-R), and adversarially chosen natural examples (ImageNet-A). These results suggest that this approach can help improve model performance under distribution shift scenarios without requiring additional assumptions about training processes or access to multiple test points. Overall, this study provides a promising direction for improving deep learning models' robustness at test time through simple yet effective techniques such as data augmentation and entropy minimization.