# UniSim: A Neural Closed-Loop Sensor Simulator

**Authors:** Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, Raquel Urtasun  
**arXiv:** [2308.01898](https://arxiv.org/abs/2308.01898)  
**Date:** 2023-08-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.01898v1/

## Key points

- Development of safe self-driving vehicles relies on rigorous testing of autonomy systems
- Collecting enough real-world data for all safety-critical scenarios is challenging
- Closed-loop simulations are necessary to accurately evaluate SDV performance
- UniSim is a neural sensor simulator that converts recorded logs into realistic closed-loop multi-sensor simulations
- UniSim reconstructs static background and dynamic actors using neural feature grids
- It composites elements together to simulate LiDAR and camera data from new viewpoints
- UniSim incorporates learnable priors for dynamic objects and uses a convolutional network to complete unseen regions
- Experiments show that UniSim generates realistic sensor data with minimal domain gap on downstream tasks
- UniSim enables closed-loop evaluation of autonomy systems on safety-critical scenarios as if in the real world
- UniSim addresses limitations of relying solely on real-world data collection
- UniSim offers a solution for generating diverse and realistic simulated scenarios for evaluating SDV performance and safety.

## Summary

The development of safe self-driving vehicles (SDVs) relies heavily on rigorous testing of autonomy systems. However, collecting enough real-world data to cover all possible safety-critical scenarios is challenging, as many of these scenarios occur rarely on public roads. To accurately evaluate the performance of SDVs, it is necessary to test them in closed-loop simulations where the vehicle and other actors interact with each other at each timestep. In this paper, the authors propose UniSim, a neural sensor simulator that converts a single recorded log captured by a sensor-equipped vehicle into a realistic closed-loop multi-sensor simulation. By using previously recorded driving logs as a resource, UniSim reconstructs both the static background and dynamic actors in the scene using neural feature grids. It then composites these elements together to simulate LiDAR and camera data from new viewpoints, accounting for changes such as adding or removing actors and altering their trajectories. To handle extrapolated views more effectively, UniSim incorporates learnable priors for dynamic objects and utilizes a convolutional network to complete unseen regions. The experiments conducted demonstrate that UniSim can generate realistic sensor data with minimal domain gap on downstream tasks. With the help of UniSim, the authors showcase closed-loop evaluation of an autonomy system on safety-critical scenarios as if it were operating in the real world. This approach addresses the limitations of relying solely on real-world data collection and enables more comprehensive testing of SDVs. Overall, UniSim offers a promising solution for generating diverse and realistic simulated scenarios for evaluating autonomous driving systems' performance and safety.