# DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving

**Authors:** Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiwen Lu  
**arXiv:** [2309.09777](https://arxiv.org/abs/2309.09777)  
**Date:** 2023-09-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.09777v1/

## Key points

- World models are popular in autonomous driving for understanding and navigating complex driving environments.
- Existing world models are often based on gaming or simulated settings, limiting their applicability to real-world scenarios.
- DriveDreamer is a novel world model derived entirely from real-world driving data.
- DriveDreamer uses a diffusion model to construct a comprehensive representation of the complex environment, capturing structural constraints and nuances of real-world traffic scenarios.
- DriveDreamer has a two-stage training pipeline: acquiring an understanding of structured traffic constraints and anticipating future states.
- DriveDreamer performs well on the nuScenes benchmark, generating precise and controllable video generation that captures the complexities of real-world traffic scenarios.
- DriveDreamer can generate realistic and reasonable driving policies, opening up new possibilities for interaction and practical applications in autonomous driving.
- DriveDreamer addresses limitations of existing research by relying on real-world driving data, allowing for more accurate representations of complex driving scenarios.

## Summary

World models have become increasingly popular in the field of autonomous driving due to their ability to understand and navigate complex driving environments. However, most existing world models are based on gaming environments or simulated settings, which limits their applicability to real-world driving scenarios. In this paper, the authors introduce DriveDreamer, a novel world model that is entirely derived from real-world driving data. To address the challenge of modeling intricate driving scenes, the authors propose using a powerful diffusion model to construct a comprehensive representation of the complex environment. This approach allows DriveDreamer to capture the structural constraints and nuances of real-world traffic scenarios. The authors also present a two-stage training pipeline for DriveDreamer. In the initial phase, the model acquires a deep understanding of structured traffic constraints, enabling it to navigate safely within these constraints. In the subsequent stage, DriveDreamer is equipped with the ability to anticipate future states, further enhancing its performance in real-world driving situations. To evaluate DriveDreamer's performance, the authors instantiate it on the challenging nuScenes benchmark and conduct extensive experiments. The results demonstrate that DriveDreamer enables precise and controllable video generation that faithfully captures the complexities of real-world traffic scenarios. Additionally, DriveDreamer is capable of generating realistic and reasonable driving policies, opening up new possibilities for interaction and practical applications in autonomous driving. Overall, DriveDreamer represents a significant advancement in world modeling for autonomous driving by addressing the limitations of existing research focused on gaming environments or simulations. Its reliance on real-world driving data allows for more accurate representations of complex driving scenarios and paves the way for safer and more efficient autonomous vehicles in the future.