# Learning Controls Using Cross-Modal Representations: Bridging Simulation and Reality for Drone Racing

**Authors:** Rogerio Bonatti, Ratnesh Madaan, Vibhav Vineet, Sebastian Scherer, Ashish Kapoor  
**arXiv:** [1909.06993](https://arxiv.org/abs/1909.06993)  
**Date:** 2019-09-16  
**Summary page:** https://www.summarizepaper.com/en/paper/1909.06993v1/

## Key points

- Machine learning still has a long way to go in solving open-world perception-control tasks
- Advancements in reinforcement and imitation learning have shown promise but require large amounts of difficult-to-collect real-world data
- Authors propose a novel approach that uses unsupervised data and simulated labeled data from FPV drone footage
- This approach combines raw camera sensor data with system states to create a joint low-dimensional representation learned through Variational Auto Encoders
- Experiments show the effectiveness of this approach in improving control policy performance and generalizing across simulation and reality
- The work highlights challenges in developing robust solutions for open-world perception-control tasks like FPV drone racing
- The proposed solution utilizes cross-modal architecture and unsupervised data

## Summary

, , , , 

In the field of machine learning, there is still a long way to go in terms of developing machines that can effectively solve open-world perception-control tasks, such as first-person view (FPV) drone racing. Recent advancements in reinforcement and imitation learning have shown promise, but are limited by the need for large amounts of real-world data that is difficult to collect. To address this issue, the authors propose a novel approach that leverages unsupervised data and simulated labeled data from FPV drone footage. This cross-modal perspective combines raw camera sensor data with system states relevant to the task, resulting in a joint low-dimensional representation learned through Variational Auto Encoders. The experiments conducted demonstrate the effectiveness of this approach in improving control policy performance and its ability to generalize across simulation and reality. Overall, this work highlights the challenges faced in developing robust solutions for open-world perception-control tasks like FPV drone racing and introduces a promising solution using cross-modal architecture and unsupervised data.