# You Only Segment Once: Towards Real-Time Panoptic Segmentation

**Authors:** Jie Hu, Linyan Huang, Tianhe Ren, Shengchuan Zhang, Rongrong Ji, Liujuan Cao  
**arXiv:** [2303.14651](https://arxiv.org/abs/2303.14651)  
**Date:** 2023-03-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.14651v1/

## Key points

- YOSO is a real-time panoptic segmentation framework
- It aims to achieve efficiency and accuracy in image segmentation
- YOSO predicts masks using dynamic convolutions between panoptic kernels and image feature maps
- It enables segmentation of both instance and semantic tasks with one pass
- The authors designed a feature pyramid aggregator for efficient feature map extraction
- They also designed a separable dynamic decoder for panoptic kernel generation
- YOSO performs multi-head cross-attention through separable dynamic convolution to enhance efficiency and accuracy
- YOSO achieves competitive performance compared to state-of-the-art models
- Impressive results on various datasets: 46.4 PQ at 45.6 FPS on COCO, 52.5 PQ at 22.6 FPS on Cityscapes, 38.0 PQ at 35.4 FPS on ADE20K, and 34.1 PQ at 7.1 FPS on Mapillary Vistas.
- Increasing the number of stages improves PQ performance but decreases FPS performance; two stages strike the best balance between speed and accuracy.
- Increasing the number of proposal kernels from 50 to 100 improves PQ performance; saturates at 150 kernels; higher numbers decrease speed as well.

## Summary

In this paper, the authors propose YOSO, a real-time panoptic segmentation framework that aims to achieve both efficiency and accuracy in segmenting images. YOSO predicts masks by utilizing dynamic convolutions between panoptic kernels and image feature maps, enabling the segmentation of both instance and semantic tasks with just one pass. To reduce computational overhead, the authors design a feature pyramid aggregator for efficient feature map extraction and a separable dynamic decoder for panoptic kernel generation. The feature pyramid aggregator re-parameterizes interpolation-first modules in a convolution-first manner, significantly speeding up the pipeline without any additional costs. Meanwhile, the separable dynamic decoder performs multi-head cross-attention through separable dynamic convolution to enhance both efficiency and accuracy. One notable aspect of YOSO is its competitive performance compared to state-of-the-art models. It achieves impressive results on various datasets, including 46.4 PQ at 45.6 FPS on COCO, 52.5 PQ at 22.6 FPS on Cityscapes, 38.0 PQ at 35.4 FPS on ADE20K, and 34.1 PQ at 7.1 FPS on Mapillary Vistas. The authors also explore different configurations to optimize YOSO's performance further. They find that increasing the number of stages improves PQ performance but decreases FPS performance; after experimentation they determine that using two stages strikes the best balance between speed and accuracy. Additionally, they investigate the impact of the number of proposal kernels on performance and find that increasing it from 50 to 100 improves PQ performance while saturating at 150 kernels; however higher numbers of proposal kernels decrease speed as well.