# Virtual Worlds as Proxy for Multi-Object Tracking Analysis

**Authors:** Adrien Gaidon, Qiao Wang, Yohann Cabon, Eleonora Vig  
**arXiv:** [1605.06457](https://arxiv.org/abs/1605.06457)  
**Date:** 2016-05-20  
**Summary page:** https://www.summarizepaper.com/en/paper/1605.06457v1/

## Key points

- Development of accurate computer vision algorithms often requires expensive data and manual labeling
- Advancement in computer graphics allows for generating fully labeled virtual worlds that are dynamic and photo-realistic
- Authors propose an efficient method for cloning real-world scenarios into virtual environments
- Video dataset called Virtual KITTI is created, automatically labeled with ground truth information for various tasks
- Quantitative experiments compare behavior of deep learning algorithms trained on real data vs virtual data
- Algorithms exhibit similar performance in both real and virtual worlds, pre-training on virtual data can improve performance
- Virtual worlds allow researchers to measure impact of weather conditions and imaging settings on recognition performance
- Proxy virtual worlds can be effective substitutes for real-world data acquisition and labeling in computer vision research

## Summary

In the field of computer vision, the development of accurate algorithms often relies on acquiring expensive data and manually labeling it. However, a recent advancement in computer graphics has opened up new possibilities for generating fully labeled virtual worlds that are dynamic and photo-realistic. In this study, the authors leverage this progress to propose an efficient method for cloning real-world scenarios into virtual environments. To validate their approach, the authors have created a video dataset called Virtual KITTI. This dataset is automatically labeled with ground truth information for various tasks such as object detection, tracking, scene segmentation, instance segmentation, depth estimation, and optical flow. By releasing this dataset to the public, they provide a valuable resource for researchers in the field. The authors also conduct quantitative experiments to compare the behavior of modern deep learning algorithms when trained on real data versus virtual data. They find that these algorithms exhibit similar performance in both real and virtual worlds. Furthermore, they observe that pre-training on virtual data can actually improve algorithm performance. One significant advantage of using virtual worlds is that they allow researchers to measure the impact of different weather conditions and imaging settings on recognition performance. By keeping all other factors equal, researchers can isolate the effects of these variables on deep models for tracking. Overall, this study demonstrates how proxy virtual worlds can serve as effective substitutes for real-world data acquisition and labeling in computer vision research. The findings suggest that pre-training on virtual data can enhance algorithm performance and highlight the importance of considering environmental factors when developing tracking models.