# Scale-Aware Modulation Meet Transformer

**Authors:** Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, Lianwen Jin  
**arXiv:** [2307.08579](https://arxiv.org/abs/2307.08579)  
**Date:** 2023-07-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.08579v1/

## Key points

- Scale-Aware Modulation Transformer (SMT) combines convolutional networks and vision Transformers for efficient handling of downstream tasks
- SMT introduces the Multi-Head Mixed Convolution (MHMC) module to capture multi-scale features and expand receptive field
- SMT also includes the Scale-Aware Aggregation (SAA) module for lightweight but effective information fusion across different heads
- Evolutionary Hybrid Network (EHN) simulates the shift from capturing local to global dependencies as network depth increases, resulting in superior performance
- SMT outperforms existing state-of-the-art models across various visual tasks
- Achieves high top-1 accuracy on ImageNet-1K with different model sizes and computational costs
- Pretrained on ImageNet-22K, achieves high top-1 accuracy when finetuned with different resolutions
- SMT base outperforms Swin Transformer in object detection using Mask R-CNN on COCO dataset, both with 1x and 3x training schedules.

## Summary

This paper introduces a new vision Transformer called Scale-Aware Modulation Transformer (SMT) that combines convolutional networks and vision Transformers to efficiently handle various downstream tasks. The proposed Scale-Aware Modulation (SAM) in SMT consists of two primary novel designs. Firstly, the Multi-Head Mixed Convolution (MHMC) module is introduced to capture multi-scale features and expand the receptive field. Secondly, the Scale-Aware Aggregation (SAA) module enables lightweight but effective information fusion across different heads. These modules enhance convolutional modulation. In contrast to previous works that utilized modulations throughout all stages to build an attention-free network, this paper proposes an Evolutionary Hybrid Network (EHN). EHN effectively simulates the shift from capturing local to global dependencies as the network becomes deeper, resulting in superior performance. Extensive experiments demonstrate that SMT outperforms existing state-of-the-art models across a wide range of visual tasks. For instance, SMT achieves 82.2% and 84.3% top-1 accuracy on ImageNet-1K with 11.5M / 2.4GFLOPs and 32M / 7.7GFLOPs, respectively. After being pretrained on ImageNet-22K in 224^2 resolution, it attains 87.1% and 88.1% top-1 accuracy when finetuned with resolutions of 224^2 and 384^2, respectively. For object detection using Mask R-CNN, the SMT base trained with a 1x schedule outperforms the Swin Transformer counterpart by 4.2 mAP on COCO dataset, while training with a 3x schedule surpasses it by 1.3 mAP.