# Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing

**Authors:** Magnus Wrenninge, Jonas Unger  
**arXiv:** [1810.08705](https://arxiv.org/abs/1810.08705)  
**Date:** 2018-10-19  
**Summary page:** https://www.summarizepaper.com/en/paper/1810.08705v1/

## Key points

- Synscapes is a synthetic dataset for street scene parsing created using photorealistic rendering techniques
- State-of-the-art results achieved by training and validating on Synscapes dataset
- Analysis of networks trained on real data performing inference on synthetic data to determine equivalence of simulation environments
- Comparison of performance between networks trained on synthetic data and evaluated on real-world data
- Correlations between scenario parameters used to generate synthetic data and segmentation network's performance examined using linear regression and IoU scores
- Insights into existing organically-trained neural network gained by leveraging Synscapes' images, annotations, and metadata as testing proxy
- Future work could focus on understanding differences in realism and sensor simulation fidelity choices during synthetic data generation
- Establishing metrics for evaluating simulation quality such as geometric variation, textural accuracy, material physicality, and illumination complexity
- Realism plays a significant role in making simulation a core component of machine learning and computer vision systems

## Summary

In this study, we present Synscapes, a synthetic dataset for street scene parsing that is created using photorealistic rendering techniques. We demonstrate the state-of-the-art results achieved by training and validating on this dataset, as well as explore new types of analysis enabled by its detailed metadata. One of the key aspects we investigate is the behavior of networks trained on real data when performing inference on synthetic data. This analysis is crucial in determining the equivalence of simulation environments. We also compare the performance of networks trained on synthetic data and evaluated on real-world data. To illustrate how Synscapes' detailed metadata and broad variation can be utilized, we examine correlations between scenario parameters used to generate the synthetic data and a segmentation network's performance. By dividing predictions into subsets based on parameter selection and computing Intersection over Union (IoU) scores for each class, we perform linear regression to identify correlations. Pairs with p-values below 0.05 are discarded, revealing interesting insights about the relationship between scenario parameters and segmentation performance. Furthermore, we leverage Synscapes' unique images, annotations, and metadata to gain insights into an existing organically-trained neural network. By using the synthetic data as a richly-annotated testing proxy, we uncover correlations and biases in networks trained on Cityscapes and KITTI datasets. Future work could delve deeper into understanding the differences in realism and sensor simulation fidelity choices made during synthetic data generation. Establishing metrics for geometric variation, textural accuracy, material physicality, and illumination complexity would be valuable in evaluating simulation quality. Realism appears to play a significant role in making simulation a core component of machine learning and computer vision systems. Overall, our study demonstrates how Synscapes provides opportunities for fine-grained analysis of computer vision systems by offering uncorrelated images with detailed annotations.