# Fully Self-Supervised Learning for Semantic Segmentation

**Authors:** Yuan Wang, Wei Zhuo, Yucong Li, Zhi Wang, Qi Ju, Wenwu Zhu  
**arXiv:** [2202.11981](https://arxiv.org/abs/2202.11981)  
**Date:** 2022-02-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2202.11981v1/

## Key points

- Wang et al. propose a fully self-supervised framework for semantic segmentation called FS^4
- The authors emphasize the importance of a bootstrapped strategy for semantic segmentation
- Bootstrapped strategy reduces the need for annotation and enables customized models for open-world domains
- Recent self-supervised methods are dependent on fully supervised pretrained models, limiting their self-supervision capabilities
- Authors introduce a bootstrapped training scheme using Pyramid-Global-Guided (PGG) strategy and Context-Aware Embedding (CAE) module
- PGG training strategy involves supervising learning with pyramid image/patch level pseudo labels generated by grouping unsupervised features
- CAE module generates global feature embeddings considering neighbors close in space and appearance
- Proposed method evaluated on COCO Stuff dataset, shows significant improvements compared to existing approaches (+7.19 mIoU)
- Framework addresses limitations by leveraging global semantic knowledge and context aware embeddings

## Summary

In this work, Wang et al. propose a fully self-supervised framework for semantic segmentation called FS^4. The authors emphasize the importance of a bootstrapped strategy for semantic segmentation as it reduces the need for annotation and enables the construction of customized models for open-world domains. This application is especially important in realistic scenarios. While recent self-supervised semantic segmentation methods have made significant progress, they are heavily dependent on fully supervised pretrained models, making it impossible to achieve a fully self-supervised pipeline. To address this limitation, the authors introduce a bootstrapped training scheme that leverages global semantic knowledge for self-supervision using their proposed Pyramid-Global-Guided (PGG) strategy and Context-Aware Embedding (CAE) module. The PGG training strategy involves supervising learning with pyramid image/patch level pseudo labels generated by grouping unsupervised features. These stable global and pyramid semantic pseudo labels prevent the segmentation model from learning excessive clutter regions or degrading to one background region. Additionally, the CAE module generates global feature embeddings considering neighbors close both in space and appearance in a non-trivial way. This context aware embedding enhances the overall performance of the framework. The proposed method is evaluated on the large scale COCO Stuff dataset, demonstrating significant improvements of 7.19 mIoU on both things and stuff objects compared to existing approaches. In conclusion, Wang et al. 's fully self-supervised framework addresses the limitations of previous methods by providing an effective bootstrapped training scheme that leverages global semantic knowledge and context aware embeddings. The experimental results validate its effectiveness in improving semantic segmentation performance on real world datasets.