# Global Context Vision Transformers

**Authors:** Ali Hatamizadeh, Hongxu Yin, Jan Kautz, Pavlo Molchanov  
**arXiv:** [2206.09959](https://arxiv.org/abs/2206.09959)  
**Date:** 2022-06-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.09959v4/

## Key points

- The paper proposes a novel architecture called GC ViT for computer vision tasks
- GC ViT enhances parameter and compute utilization
- The model consists of global context self-attention modules combined with standard local self-attention to effectively model both long and short-range spatial interactions
- The proposed downsampler leverages a parameter-efficient fused inverted residual block to improve the modeling of inter-channel dependencies
- GC ViT achieves new state-of-the-art performance across image classification, object detection, and semantic segmentation tasks
- Pre-trained GC ViT backbones outperform prior work consistently by large margins in downstream tasks of object detection, instance segmentation, and semantic segmentation on MS COCO and ADE20K datasets
- The success of this proposed architecture can be attributed to its ability to enhance parameter efficiency while maintaining high accuracy levels across various computer vision tasks.

## Summary

In their paper titled "Global Context Vision Transformers," Ali Hatamizadeh, Hongxu Yin, Jan Kautz, and Pavlo Molchanov propose a novel architecture called GC ViT that enhances parameter and compute utilization for computer vision tasks. The core of the model consists of global context self-attention modules combined with standard local self-attention to effectively model both long and short-range spatial interactions. This approach is an alternative to complex operations such as attention masks or local windows shifting. While the local self-attention modules are responsible for modeling short-range information, the global query tokens are shared across all global self-attention modules to interact with local key and values. The authors also address the lack of inductive bias in ViTs by proposing a novel downsampler that leverages a parameter-efficient fused inverted residual block. This improves the modeling of inter-channel dependencies. The proposed GC ViT achieves new state-of-the-art performance across image classification, object detection, and semantic segmentation tasks. On ImageNet-1K dataset for classification, GC ViT models with 51M, 90M and 201M parameters achieve 84.3%, 84.9% and 85.6% Top-1 accuracy respectively, surpassing comparably sized prior art such as CNN based ConvNeXt and ViT based Swin Transformer. Moreover, pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation and semantic segmentation on MS COCO and ADE20K datasets outperform prior work consistently by large margins. The success of this proposed architecture can be attributed to its ability to enhance parameter efficiency while maintaining high accuracy levels across various computer vision tasks. Overall, this research provides valuable insights into improving transformer based models' performance in computer vision applications.