# Revisiting ResNets: Improved Training and Scaling Strategies

**Authors:** Irwan Bello, William Fedus, Xianzhi Du, Ekin D. Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, Barret Zoph  
**arXiv:** [2103.07579](https://arxiv.org/abs/2103.07579)  
**Date:** 2021-03-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.07579v1/

## Key points

- Authors Irwan Bello, William Fedus, Xianzhi Du, Ekin D. Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph focus on the ResNet model in their paper "Revisiting ResNets: Improved Training and Scaling Strategies."
- Research indicates that training and scaling strategies may be more significant than architectural modifications in enhancing model performance.
- Introduce two novel scaling strategies: depth scaling for overfitting scenarios and width scaling for other cases; slower increase in image resolution compared to previous recommendations.
- Development of a new family of ResNet architectures called ResNet-RS with impressive performance metrics surpassing EfficientNets on TPUs while achieving comparable accuracies on ImageNet.
- In large-scale semi-supervised learning setup, ResNet-RS achieves top-1 ImageNet accuracy of 86.2%, outperforming EfficientNet NoisyStudent by being 4.7x faster.
- Enhanced transfer performance across various downstream tasks and video classification tasks on Kinetics-400 using refined ResNets as baseline models recommended by authors for future research endeavors.

## Summary

In their paper titled "Revisiting ResNets: Improved Training and Scaling Strategies," authors Irwan Bello, William Fedus, Xianzhi Du, Ekin D. Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph delve into the realm of computer vision architectures with a focus on the canonical ResNet model. The study aims to dissect the impact of model architecture changes versus alterations in training methodology and scaling strategies. Surprisingly, the research reveals that training and scaling strategies may hold more significance than architectural modifications. The resulting ResNets not only match recent state-of-the-art models but also outperform them in certain aspects. The authors introduce two novel scaling strategies based on their findings: one suggests scaling model depth in scenarios where overfitting is likely to occur while width scaling is preferred in other cases; the second strategy advises a slower increase in image resolution compared to previous recommendations by Tan & Le (2019). By implementing these improved training and scaling techniques, the team develops a new family of ResNet architectures known as ResNet-RS. These models exhibit impressive performance metrics being 1.7x - 2.7x faster than EfficientNets on TPUs while achieving comparable accuracies on ImageNet. In a large-scale semi-supervised learning setup, ResNet-RS achieves an impressive top-1 ImageNet accuracy of 86.2%, surpassing EfficientNet NoisyStudent by being 4.7x faster. Furthermore, the training techniques employed enhance transfer performance across various downstream tasks rivaling state-of-the-art self-supervised algorithms. The benefits extend to video classification tasks on Kinetics-400 as well. The authors recommend practitioners adopt these refined ResNets as baseline models for future research endeavors due to their simplicity yet superior performance outcomes compared to existing alternatives. Overall, this study sheds light on the importance of training methodology and scaling strategies in optimizing deep learning architectures for computer vision applications.