AI summary · arXiv: 1903.11027v5 · cs.LG
nuScenes: A multimodal dataset for autonomous driving
The essentials
- Robust detection and tracking of objects are essential for safe and efficient operation of autonomous vehicles
- Autonomous vehicles are equipped with a combination of cameras, lidar, and radar sensors
- The authors present the nuScenes dataset, which includes 6 cameras, 5 radars, and 1 lidar providing a complete 360-degree field of view
- The dataset comprises 1000 scenes with detailed annotations for 23 different classes of objects along with 8 attributes
- nuScenes dataset offers seven times more annotations and a hundred times more images compared to the KITTI dataset
- Introduces novel metrics for evaluating 3D object detection and tracking performance
- Provides baseline results for both lidar-based and image-based detection and tracking methods
- Researchers can access the data online to facilitate advancements in autonomous driving technology
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 31 papers around this one, the name that comes back most: Andreas Geiger (2 papers)
3 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous DrivingLian-Qing Zheng, Long Yang, Qunshu Lin et al. · 2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cited 36 times · Not summarized yet
OmniHD-Scenes introduces a multimodal autonomous-driving dataset with broad sensor coverage, as nuScenes does with cameras, radar, and lidar.
Abstract
The rapid advancement of deep learning has intensified the need for comprehensive data for use by autonomous driving algorithms. High-quality datasets are crucial for the development of effective data-driven autonomous driving solutions. Next-generation autonomous driving datasets must be multimodal, incorporating data from advanced sensors that feature extensive data coverage, detailed annotations, and diverse scene representation. To address this need, we present OmniHD-Scenes, a large-scale multimodal dataset that provides comprehensive omnidirectional high-definition data. The OmniHD-Scenes dataset combines data from 128-beam LiDAR, six cameras, and six 4D imaging radar systems to achiev…
-
nuScenes Revisited: Progress and Challenges in Autonomous DrivingWhye Kit Fong, Venice Erin Liong, K. Tan et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet
nuScenes Revisited examines the nuScenes dataset, discussing its role and challenges in autonomous-vehicle development.
Abstract
Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS) have been revolutionized by Deep Learning. As a data-driven approach, Deep Learning relies on vast amounts of driving data, typically labeled in great detail. As a result, datasets, alongside hardware and algorithms, are foundational building blocks for the development of AVs. In this work we revisit one of the most widely used autonomous driving datasets: the nuScenes dataset. nuScenes exemplifies key trends in AV development, being the first dataset to include radar data, to feature diverse urban driving scenes from two continents, and to be collected using a fully autonomous vehicle operating on public roads, while als…
-
aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range PerceptionTamás Matuszka, Iván Barton, Ádám Butykai et al. · 2022 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 44 times · AI summary available
aiMotive introduces a multimodal autonomous-driving dataset, emphasizing robustness and long-range perception missing from existing datasets.
Abstract
Autonomous driving is a popular research area within the computer vision research community. Since autonomous vehicles are highly safety-critical, ensuring robustness is essential for real-world deployment. While several public multimodal datasets are accessible, they mainly comprise two sensor modalities (camera, LiDAR) which are not well suited for adverse weather. In addition, they lack far-range annotations, making it harder to train neural networks that are the base of a highway assistant function of an autonomous vehicle. Therefore, we introduce a multimodal dataset for robust autonomous driving with long-range perception. The dataset consists of 176 scenes with synchronized and calibr…
-
A Unified Benchmark for Aerial-Enhanced Autonomous Driving: The KITScenes Aerial DatasetDaniel Lengerer, Richard Schwarzkopf, Jonas Merkert et al. · 2026 · Not summarized yet
KITScenes Aerial combines onboard driving data with aerial imagery to benchmark mapping and cross-view tasks, rather than introducing the sensor dataset nuScenes presents.
Abstract
Aerial imagery provides complementary spatial context for autonomous driving by exposing road structure beyond the immediate perception range of onboard sensors. Yet map construction from aerial imagery, aerial-assisted online HD mapping, and cross-view localization are typically studied on separate data and benchmarks. We introduce KITScenes Aerial, an extension of the KITScenes Multimodal dataset that augments synchronized onboard sensor data, vehicle poses, and rich Lanelet2 map annotations with spatially associated aerial imagery and higher-resolution neural representations, providing a common basis for a variety of research tasks. Our mapping evaluation focuses on all 53 KITScenes map c…
-
MAN TruckScenes: A multimodal dataset for autonomous trucking in diverse conditionsF. Fent, Fabian Kuttenreich, Florian Ruch et al. · 2024 · Neural Information Processing Systems · cited 65 times · Not summarized yet
MAN TruckScenes presents a multimodal dataset for autonomous trucking, targeting truck-specific perception challenges.
Abstract
Autonomous trucking is a promising technology that can greatly impact modern logistics and the environment. Ensuring its safety on public roads is one of the main duties that requires an accurate perception of the environment. To achieve this, machine learning methods rely on large datasets, but to this day, no such datasets are available for autonomous trucks. In this work, we present MAN TruckScenes, the first multimodal dataset for autonomous trucking. MAN TruckScenes allows the research community to come into contact with truck-specific challenges, such as trailer occlusions, novel sensor perspectives, and terminal environments for the first time. It comprises more than 740 scenes of 20s…
-
NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving ScenarioTianwen Qian, Jingjing Chen, Linhai Zhuo et al. · 2023 · Computer Vision and Pattern Recognition · cited 432 times · AI summary available
NuScenes-QA uses nuScenes data for multimodal, multi-frame visual question answering rather than detection and tracking benchmarks.
Abstract
We introduce a novel visual question answering (VQA) task in the context of autonomous driving, aiming to answer natural language questions based on street-view clues. Compared to traditional VQA tasks, VQA in autonomous driving scenario presents more challenges. Firstly, the raw visual data are multi-modal, including images and point clouds captured by camera and LiDAR, respectively. Secondly, the data are multi-frame due to the continuous, real-time acquisition. Thirdly, the outdoor scenes exhibit both moving foreground and static background. Existing VQA benchmarks fail to adequately address these complexities. To bridge this gap, we propose NuScenes-QA, the first benchmark for VQA in the…
-
A*3D Dataset: Towards Autonomous Driving in Challenging EnvironmentsQuang-Hieu Pham, P. Sevestre, R. Pahwa et al. · 2019 · IEEE International Conference on Robotics and Automation · cited 190 times · Not summarized yet
A*3D introduces an RGB-and-LiDAR driving dataset with varied weather, scenes, and nighttime data for computer vision benchmarking.
Abstract
With the increasing global popularity of self-driving cars, there is an immediate need for challenging real-world datasets for benchmarking and training various computer vision tasks such as 3D object detection. Existing datasets either represent simple scenarios or provide only day-time data. In this paper, we introduce a new challenging A*3D dataset which consists of RGB images and LiDAR data with a significant diversity of scene, time, and weather. The dataset consists of high-density images (≈ 10 times more than the pioneering KITTI dataset), heavy occlusions, a large number of nighttime frames (≈ 3 times the nuScenes dataset), addressing the gaps in the existing datasets to push the bou…
-
One Million Scenes for Autonomous Driving: ONCE DatasetJiageng Mao, Minzhe Niu, Chenhan Jiang et al. · 2021 · NeurIPS Datasets and Benchmarks · cited 402 times · Not summarized yet
ONCE introduces a large autonomous-driving dataset to support perception research and learning from unlabeled scenes.
Abstract
Current perception models in autonomous driving have become notorious for greatly relying on a mass of annotated data to cover unseen cases and address the long-tail problem. On the other hand, learning from unlabeled large-scale collected data and incrementally self-training powerful recognition models have received increasing attention and may become the solutions of next-generation industry-level powerful and robust perception models in autonomous driving. However, the research community generally suffered from data inadequacy of those essential real-world scene data, which hampers the future exploration of fully/semi/self-supervised methods for 3D perception. In this paper, we introduce …
Prior work it builds on
-
The ApolloScape Open Dataset for Autonomous Driving and Its ApplicationXin-Yu Huang, Peng Wang, Xinjing Cheng et al. · 2018 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cited 692 times · Not summarized yet
ApolloScape presents a large labeled autonomous-driving dataset for training and evaluating perception systems.
Abstract
Autonomous driving has attracted tremendous attention especially in the past few years. The key techniques for a self-driving car include solving tasks like 3D map construction, self-localization, parsing the driving road and understanding objects, which enable vehicles to reason and act. However, large scale data set for training and system evaluation is still a bottleneck for developing robust perception models. In this paper, we present the ApolloScape dataset [1] and its applications for autonomous driving. Compared with existing public datasets from real scenes, e.g., KITTI [2] or Cityscapes [3] , ApolloScape contains much large and richer labelling including holistic semantic dense poi…
-
The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban ScenesAbhishek Patil, Srikanth Malla, Hai-Ming Gang et al. · 2019 · IEEE International Conference on Robotics and Automation · cited 265 times · Not summarized yet
H3D provides full-surround LiDAR data and labels for 3D multi-object detection and tracking in crowded traffic.
Abstract
3D multi-object detection and tracking are crucial for traffic scene understanding. However, the community pays less attention to these areas due to the lack of a standardized benchmark dataset to advance the field. Moreover, existing datasets (e.g., KITTI [1]) do not provide sufficient data and labels to tackle challenging scenes where highly interactive and occluded traffic participants are present. To address the issues, we present the Honda Research Institute 3D Dataset (H3D), a large-scale full-surround 3D multi-object detection and tracking dataset collected using a 3D LiDAR scanner. H3D comprises of 160 crowded and highly interactive traffic scenes with a total of 1 million labeled in…
-
The Cityscapes Dataset for Semantic Urban Scene UnderstandingMarius Cordts, Mohamed Omran, Sebastian Ramos et al. · 2016 · Computer Vision and Pattern Recognition · cited 14,475 times · Not summarized yet
Cityscapes introduces urban driving imagery for pixel- and instance-level semantic labeling, another benchmark dataset for scene understanding.
Abstract
Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotation…
-
KAIST Multi-Spectral Day/Night Data Set for Autonomous and Assisted DrivingYukyung Choi, Namil Kim, Soonmin Hwang et al. · 2018 · IEEE transactions on intelligent transportation systems (Print) · cited 362 times · Not summarized yet
KAIST provides multispectral day-and-night driving data, addressing sensor and lighting conditions adjacent to nuScenes' multimodal dataset focus.
-
Probabilistic 3D Multi-Object Tracking for Autonomous DrivingHsu-kuang Chiu, Antonio Prioletti, Jie Li et al. · 2020 · arXiv.org · preprint, not peer reviewed · cited 154 times · Not summarized yet
Probabilistic 3D Multi-Object Tracking presents a Kalman-filter method that won the nuScenes Tracking Challenge.
Abstract
3D multi-object tracking is a key module in autonomous driving applications that provides a reliable dynamic representation of the world to the planning module. In this paper, we present our on-line tracking method, which made the first place in the NuScenes Tracking Challenge, held at the AI Driving Olympics Workshop at NeurIPS 2019. Our method estimates the object states by adopting a Kalman Filter. We initialize the state covariance as well as the process and observation noise covariance with statistics from the training set. We also use the stochastic information from the Kalman Filter in the data association step by measuring the Mahalanobis distance between the predicted object states …
Papers that cite it
-
Scalability in Perception for Autonomous Driving: Waymo Open DatasetPei Sun, Henrik Kretzschmar, Xerxes Dotiwalla et al. · 2019 · Computer Vision and Pattern Recognition · cited 4,635 times · Not summarized yet
Waymo Open Dataset introduces synchronized, calibrated lidar and camera driving data for research on scalable autonomous-vehicle perception.
Abstract
The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the over-all viability of the technology. In an effort to help align the research community’s contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data capt…
-
LynxScenes: A Large-Scale Real-World Benchmark for Outdoor Wheel-Legged Robot Perception, Navigation, Imagination and ReconstructionJun Tian, Hao-Rui Wu, Ruo Chen et al. · 2026 · Not summarized yet
LynxScenes provides synchronized multimodal data and benchmarks for outdoor wheel-legged robot tasks, rather than autonomous-car perception.
Abstract
Wheel-legged robots operating in outdoor spaces shared with humans require accurate perception, socially compliant navigation, future-scene reasoning, and scene reconstruction. However, existing datasets typically support these capabilities in isolation and rarely provide dense spatial and behavior-level supervision on real wheel-legged platforms. We introduce LynxScenes, a large-scale real-world dataset and benchmark collected with a LYNX M20 wheel-legged robot. It contains approximately 100K 30-s navigation clips spanning 833.6 h and 3,294.7 km, with 3.0M annotated frames. The synchronized recordings include four fisheye cameras, dual LiDARs, IMU, odometry, and robot states. Annotations co…
-
PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous DrivingYi Feng, Zi-Zhan Guo, E. Junwu et al. · 2026 · Not summarized yet
PanoBench includes a nuScenes branch with unified interfaces and evaluation protocols for panoptic occupancy prediction.
Abstract
Panoptic occupancy unifies 3D geometry, semantics, and instance identities, but its development is limited by scarce instance-aware annotations. Existing benchmarks mainly provide semantic labels reconstructed from incomplete sensor observations, while panoptic datasets have limited domain coverage or annotation completeness. We introduce PanoBench, a camera-centric panoptic occupancy prediction benchmark spanning synthetic and real-world driving scenes. It combines PanoBench-CarlaOcc, PanoBench-nuScenes, and PanoBench-KITTI360 through canonical data interfaces and unified evaluation protocols, providing multi-resolution annotations with persistent instance identities. The synthetic branch d…
-
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and ChallengesDi Feng, Christian Haase-Schütz, Lars Rosenbaum et al. · 2019 · IEEE transactions on intelligent transportation systems (Print) · cited 1,423 times · Not summarized yet
This review surveys multimodal detection and segmentation methods, datasets, and challenges for autonomous-driving perception.
Abstract
Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detect…
-
Adversarial Robustness of Camera-Radar Sensor Fusion for Autonomous DrivingCe Zhou, Yu-Bo Wu, K. Nguyen et al. · 2026 · Not summarized yet
This study evaluates camera-radar fusion robustness on nuScenes for 3D detection and depth estimation under simulated attacks.
Abstract
Autonomous driving (AD) relies on multi-sensor perception to interpret the surrounding environment. However, the robustness of camera--radar fusion under adversarial conditions remains largely unexplored. This paper presents a systematic measurement study of the robustness of camera--radar sensor fusion in AD. We construct a unified evaluation pipeline on the nuScenes dataset that simulates attacks on both sensing modalities, including eight camera-image perturbations and three radar point-cloud perturbations. We evaluate five camera--radar fusion algorithms across two perception tasks, 3D object detection and depth estimation, and quantify their performance under clean and attacked inputs u…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 1 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.