# Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes

**Authors:** Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, Carsten Rother  
**arXiv:** [1708.01566](https://arxiv.org/abs/1708.01566)  
**Date:** 2017-08-04  
**Summary page:** https://www.summarizepaper.com/en/paper/1708.01566v1/

## Key points

- Deep learning in computer vision relies on large annotated datasets
- Virtually rendered 3D worlds are an alternative to hand-labeled images, but require significant human effort
- Authors propose a novel approach that combines real-world imagery with virtual objects to learn semantic instance segmentation and object detection models
- Method requires only a few user interactions and 3D shapes of the target object, making it more efficient than modeling complete 3D environments
- Augmented data maximally enhances performance of instance segmentation models
- Models trained on augmented imagery generalize better than those trained on synthetic data or limited amounts of annotated real data
- Efficient procedure for augmenting real images with virtual objects to generate large-scale annotated datasets for training computer vision models without requiring complex 3D modeling efforts.

## Summary

The success of deep learning in computer vision is largely dependent on the availability of large annotated datasets. To address this issue, virtually rendered 3D worlds have gained popularity as an alternative to hand-labeled images. However, generating realistic 3D content requires significant human effort. In this paper, the authors propose a novel approach that combines real-world imagery with virtual objects to learn semantic instance segmentation and object detection models. The proposed approach exploits the fact that not all aspects of a scene are equally important for these tasks and augments real-world images with virtual objects of the target category to create realistic composite images that exhibit both realistic background appearance and complex object arrangements. This method requires only a few user interactions in combination with 3D shapes of the target object, making it more efficient than modeling complete 3D environments. Through extensive experimentation, the authors determine the right set of parameters to produce augmented data that maximally enhances the performance of instance segmentation models. They demonstrate its utility by training standard deep models for semantic instance segmentation and object detection of cars in outdoor driving scenes. To test their models' generalizability, they evaluate them on two datasets: KITTI 2015 (which they annotated with pixel-accurate ground truth) and Cityscapes dataset. Their experiments show that models trained on augmented imagery generalize better than those trained on synthetic data or limited amounts of annotated real data. Overall, this work presents an efficient procedure for augmenting real images with virtual objects to generate large-scale annotated datasets for training computer vision models without requiring complex 3D modeling efforts.