# Learning from Simulation, Racing in Reality

**Authors:** Eugenio Chisari, Alexander Liniger, Alisa Rupenyan, Luc Van Gool, John Lygeros  
**arXiv:** [2011.13332](https://arxiv.org/abs/2011.13332)  
**Date:** 2020-11-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2011.13332v2/

## Key points

- The paper presents a reinforcement learning-based solution for autonomously racing on a miniature race car platform
- A policy trained purely in simulation using a relatively simple vehicle model, including model randomization, can be successfully transferred to the real robotic setup
- A novel policy output regularization approach and a lifted action space enables smooth actions but still aggressive race car driving
- The regularized policy outperforms the Soft Actor Critic (SAC) baseline method both in simulation and on the real car, but it is still outperformed by a Model Predictive Controller (MPC) state-of-the-art method
- Refining the policy with three hours of real-world interaction data allows the reinforcement learning policy to achieve lap times similar to the MPC controller while reducing track constraint violations by 50%
- Several techniques for improving training stability and performance such as curriculum learning, reward shaping, and data augmentation lead to faster convergence and better generalization of the learned policies
- Extensive experiments were conducted to evaluate the robustness of their approach under various conditions such as changes in lighting conditions, track layouts, and vehicle configurations demonstrating that their approach is highly adaptable and can handle different scenarios with minimal retraining.

## Summary

The paper presents a reinforcement learning-based solution for autonomously racing on a miniature race car platform. The authors demonstrate that a policy trained purely in simulation using a relatively simple vehicle model, including model randomization, can be successfully transferred to the real robotic setup. They achieve this by using a novel policy output regularization approach and a lifted action space that enables smooth actions but still aggressive race car driving. The regularized policy outperforms the Soft Actor Critic (SAC) baseline method both in simulation and on the real car, but it is still outperformed by a Model Predictive Controller (MPC) state-of-the-art method. However, refining the policy with three hours of real-world interaction data allows the reinforcement learning policy to achieve lap times similar to the MPC controller while reducing track constraint violations by 50%. The authors also introduce several techniques for improving training stability and performance such as curriculum learning, reward shaping, and data augmentation which lead to faster convergence and better generalization of the learned policies. Furthermore, they conduct extensive experiments to evaluate the robustness of their approach under various conditions such as changes in lighting conditions, track layouts, and vehicle configurations demonstrating that their approach is highly adaptable and can handle different scenarios with minimal retraining. Overall, this work provides valuable insights into how reinforcement learning can be used for autonomous racing on miniature race cars highlighting several key challenges and proposing effective solutions for addressing them. The findings have implications not only for robotics but also for other domains where autonomous control is required under dynamic and uncertain environments.