AI summary · arXiv: 1903.11027v5 · cs.LG

nuScenes: A multimodal dataset for autonomous driving

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan,
and 2 more authorsGiancarlo Baldan, Oscar Beijbom
Original paper · DOI 10.1109/cvpr42600.2020.01164 Submitted to arXiv on 26 Mar. 2019 Published in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020) CVPR 2020 camera ready incl. supplementary material License: NONEXCLUSIVE-DISTRIB 1.0

How to read this summary? Illustrated guide

The essentials

  • Robust detection and tracking of objects are essential for safe and efficient operation of autonomous vehicles
  • Autonomous vehicles are equipped with a combination of cameras, lidar, and radar sensors
  • The authors present the nuScenes dataset, which includes 6 cameras, 5 radars, and 1 lidar providing a complete 360-degree field of view
  • The dataset comprises 1000 scenes with detailed annotations for 23 different classes of objects along with 8 attributes
  • nuScenes dataset offers seven times more annotations and a hundred times more images compared to the KITTI dataset
  • Introduces novel metrics for evaluating 3D object detection and tracking performance
  • Provides baseline results for both lidar-based and image-based detection and tracking methods
  • Researchers can access the data online to facilitate advancements in autonomous driving technology

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Autonomous Vehicle Technology Object Detection Range Sensors nuTonomy Scenes Dataset Computer Vision Research

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
In the rapidly evolving field of autonomous vehicle technology, robust detection and tracking of objects are essential for safe and efficient operation. While image-based benchmark datasets have been instrumental in advancing computer vision tasks such as object detection, tracking, and segmentation, most autonomous vehicles are equipped with a combination of cameras and range sensors like lidar and radar. As machine learning methods for detection and tracking continue to gain traction, there is a growing need to train and evaluate these algorithms on datasets that incorporate both image data and range sensor information. Addressing this need, the authors present , a groundbreaking dataset that encompasses the full suite of sensors typically found on autonomous vehicles: 6 cameras, 5 radars, and 1 lidar, all offering a complete 360-degree field of view. Comprising 1000 scenes, each lasting 20 seconds, is meticulously annotated with detailed 3D bounding boxes for 23 different classes of objects along with 8 attributes. Notably, this dataset boasts seven times more annotations and a hundred times more images compared to the pioneering KITTI dataset. In addition to providing an extensive dataset for training and evaluation purposes, introduces novel metrics for evaluating 3D object detection and tracking performance. The authors also offer comprehensive dataset analysis along with baseline results for both lidar-based and image-based detection and tracking methods. Researchers and developers can access the data,,and further information online to facilitate advancements in autonomous driving technology. The paper "nuScenes: A multimodal dataset for autonomous driving" authored by Holger Caesar et al., presents a significant contribution to the field of computer vision research by introducing a comprehensive dataset that reflects real-world conditions faced by autonomous vehicles. This resource has the potential to drive innovation in object detection and tracking algorithms tailored specifically for autonomous driving applications.

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts More than 300 papers already cite it: be emailed each new one. Citation alerts guide
Who works on it?

Among the 31 papers around this one, the name that comes back most: Andreas Geiger (2 papers)

3 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.

Continue with Google Create a free account: your first map is free
Weekly letter The latest letter of its field, "Artificial intelligence this week" of 16 November 2026: Artificial intelligence this week: What impressive scores leave out · Receive one by email for free

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close Same question
    OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving
    Lian-Qing Zheng, Long Yang, Qunshu Lin et al. · 2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cited 36 times · Not summarized yet

    OmniHD-Scenes introduces a multimodal autonomous-driving dataset with broad sensor coverage, as nuScenes does with cameras, radar, and lidar.

    Abstract

    The rapid advancement of deep learning has intensified the need for comprehensive data for use by autonomous driving algorithms. High-quality datasets are crucial for the development of effective data-driven autonomous driving solutions. Next-generation autonomous driving datasets must be multimodal, incorporating data from advanced sensors that feature extensive data coverage, detailed annotations, and diverse scene representation. To address this need, we present OmniHD-Scenes, a large-scale multimodal dataset that provides comprehensive omnidirectional high-definition data. The OmniHD-Scenes dataset combines data from 128-beam LiDAR, six cameras, and six 4D imaging radar systems to achiev…

  2. Very close Same question
    nuScenes Revisited: Progress and Challenges in Autonomous Driving
    Whye Kit Fong, Venice Erin Liong, K. Tan et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet

    nuScenes Revisited examines the nuScenes dataset, discussing its role and challenges in autonomous-vehicle development.

    Abstract

    Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS) have been revolutionized by Deep Learning. As a data-driven approach, Deep Learning relies on vast amounts of driving data, typically labeled in great detail. As a result, datasets, alongside hardware and algorithms, are foundational building blocks for the development of AVs. In this work we revisit one of the most widely used autonomous driving datasets: the nuScenes dataset. nuScenes exemplifies key trends in AV development, being the first dataset to include radar data, to feature diverse urban driving scenes from two continents, and to be collected using a fully autonomous vehicle operating on public roads, while als…

  3. Very close Same question
    aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception
    Tamás Matuszka, Iván Barton, Ádám Butykai et al. · 2022 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 44 times · AI summary available

    aiMotive introduces a multimodal autonomous-driving dataset, emphasizing robustness and long-range perception missing from existing datasets.

    Abstract

    Autonomous driving is a popular research area within the computer vision research community. Since autonomous vehicles are highly safety-critical, ensuring robustness is essential for real-world deployment. While several public multimodal datasets are accessible, they mainly comprise two sensor modalities (camera, LiDAR) which are not well suited for adverse weather. In addition, they lack far-range annotations, making it harder to train neural networks that are the base of a highway assistant function of an autonomous vehicle. Therefore, we introduce a multimodal dataset for robust autonomous driving with long-range perception. The dataset consists of 176 scenes with synchronized and calibr…

  4. Very close Broader context Cites this paper
    A Unified Benchmark for Aerial-Enhanced Autonomous Driving: The KITScenes Aerial Dataset
    Daniel Lengerer, Richard Schwarzkopf, Jonas Merkert et al. · 2026 · Not summarized yet

    KITScenes Aerial combines onboard driving data with aerial imagery to benchmark mapping and cross-view tasks, rather than introducing the sensor dataset nuScenes presents.

    Abstract

    Aerial imagery provides complementary spatial context for autonomous driving by exposing road structure beyond the immediate perception range of onboard sensors. Yet map construction from aerial imagery, aerial-assisted online HD mapping, and cross-view localization are typically studied on separate data and benchmarks. We introduce KITScenes Aerial, an extension of the KITScenes Multimodal dataset that augments synchronized onboard sensor data, vehicle poses, and rich Lanelet2 map annotations with spatially associated aerial imagery and higher-resolution neural representations, providing a common basis for a variety of research tasks. Our mapping evaluation focuses on all 53 KITScenes map c…

  5. Very close Same question
    MAN TruckScenes: A multimodal dataset for autonomous trucking in diverse conditions
    F. Fent, Fabian Kuttenreich, Florian Ruch et al. · 2024 · Neural Information Processing Systems · cited 65 times · Not summarized yet

    MAN TruckScenes presents a multimodal dataset for autonomous trucking, targeting truck-specific perception challenges.

    Abstract

    Autonomous trucking is a promising technology that can greatly impact modern logistics and the environment. Ensuring its safety on public roads is one of the main duties that requires an accurate perception of the environment. To achieve this, machine learning methods rely on large datasets, but to this day, no such datasets are available for autonomous trucks. In this work, we present MAN TruckScenes, the first multimodal dataset for autonomous trucking. MAN TruckScenes allows the research community to come into contact with truck-specific challenges, such as trailer occlusions, novel sensor perspectives, and terminal environments for the first time. It comprises more than 740 scenes of 20s…

  6. Very close Same method, another problem
    NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario
    Tianwen Qian, Jingjing Chen, Linhai Zhuo et al. · 2023 · Computer Vision and Pattern Recognition · cited 432 times · AI summary available

    NuScenes-QA uses nuScenes data for multimodal, multi-frame visual question answering rather than detection and tracking benchmarks.

    Abstract

    We introduce a novel visual question answering (VQA) task in the context of autonomous driving, aiming to answer natural language questions based on street-view clues. Compared to traditional VQA tasks, VQA in autonomous driving scenario presents more challenges. Firstly, the raw visual data are multi-modal, including images and point clouds captured by camera and LiDAR, respectively. Secondly, the data are multi-frame due to the continuous, real-time acquisition. Thirdly, the outdoor scenes exhibit both moving foreground and static background. Existing VQA benchmarks fail to adequately address these complexities. To bridge this gap, we propose NuScenes-QA, the first benchmark for VQA in the…

  7. Very close Same question Cited by this paper
    A*3D Dataset: Towards Autonomous Driving in Challenging Environments
    Quang-Hieu Pham, P. Sevestre, R. Pahwa et al. · 2019 · IEEE International Conference on Robotics and Automation · cited 190 times · Not summarized yet

    A*3D introduces an RGB-and-LiDAR driving dataset with varied weather, scenes, and nighttime data for computer vision benchmarking.

    Abstract

    With the increasing global popularity of self-driving cars, there is an immediate need for challenging real-world datasets for benchmarking and training various computer vision tasks such as 3D object detection. Existing datasets either represent simple scenarios or provide only day-time data. In this paper, we introduce a new challenging A*3D dataset which consists of RGB images and LiDAR data with a significant diversity of scene, time, and weather. The dataset consists of high-density images (≈ 10 times more than the pioneering KITTI dataset), heavy occlusions, a large number of nighttime frames (≈ 3 times the nuScenes dataset), addressing the gaps in the existing datasets to push the bou…

  8. Very close Same question
    One Million Scenes for Autonomous Driving: ONCE Dataset
    Jiageng Mao, Minzhe Niu, Chenhan Jiang et al. · 2021 · NeurIPS Datasets and Benchmarks · cited 402 times · Not summarized yet

    ONCE introduces a large autonomous-driving dataset to support perception research and learning from unlabeled scenes.

    Abstract

    Current perception models in autonomous driving have become notorious for greatly relying on a mass of annotated data to cover unseen cases and address the long-tail problem. On the other hand, learning from unlabeled large-scale collected data and incrementally self-training powerful recognition models have received increasing attention and may become the solutions of next-generation industry-level powerful and robust perception models in autonomous driving. However, the research community generally suffered from data inadequacy of those essential real-world scene data, which hampers the future exploration of fully/semi/self-supervised methods for 3D perception. In this paper, we introduce …

Prior work it builds on

  1. Close Same question Used in depth by this paper A method it uses
    The ApolloScape Open Dataset for Autonomous Driving and Its Application
    Xin-Yu Huang, Peng Wang, Xinjing Cheng et al. · 2018 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cited 692 times · Not summarized yet

    ApolloScape presents a large labeled autonomous-driving dataset for training and evaluating perception systems.

    Abstract

    Autonomous driving has attracted tremendous attention especially in the past few years. The key techniques for a self-driving car include solving tasks like 3D map construction, self-localization, parsing the driving road and understanding objects, which enable vehicles to reason and act. However, large scale data set for training and system evaluation is still a bottleneck for developing robust perception models. In this paper, we present the ApolloScape dataset [1] and its applications for autonomous driving. Compared with existing public datasets from real scenes, e.g., KITTI [2] or Cityscapes [3] , ApolloScape contains much large and richer labelling including holistic semantic dense poi…

  2. Close Same question Used in depth by this paper A method it uses
    The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes
    Abhishek Patil, Srikanth Malla, Hai-Ming Gang et al. · 2019 · IEEE International Conference on Robotics and Automation · cited 265 times · Not summarized yet

    H3D provides full-surround LiDAR data and labels for 3D multi-object detection and tracking in crowded traffic.

    Abstract

    3D multi-object detection and tracking are crucial for traffic scene understanding. However, the community pays less attention to these areas due to the lack of a standardized benchmark dataset to advance the field. Moreover, existing datasets (e.g., KITTI [1]) do not provide sufficient data and labels to tackle challenging scenes where highly interactive and occluded traffic participants are present. To address the issues, we present the Honda Research Institute 3D Dataset (H3D), a large-scale full-surround 3D multi-object detection and tracking dataset collected using a 3D LiDAR scanner. H3D comprises of 160 crowded and highly interactive traffic scenes with a total of 1 million labeled in…

  3. Close Same question Used in depth by this paper A method it uses
    The Cityscapes Dataset for Semantic Urban Scene Understanding
    Marius Cordts, Mohamed Omran, Sebastian Ramos et al. · 2016 · Computer Vision and Pattern Recognition · cited 14,475 times · Not summarized yet

    Cityscapes introduces urban driving imagery for pixel- and instance-level semantic labeling, another benchmark dataset for scene understanding.

    Abstract

    Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotation…

  4. Close* Broader context A method it uses
    KAIST Multi-Spectral Day/Night Data Set for Autonomous and Assisted Driving
    Yukyung Choi, Namil Kim, Soonmin Hwang et al. · 2018 · IEEE transactions on intelligent transportation systems (Print) · cited 362 times · Not summarized yet

    KAIST provides multispectral day-and-night driving data, addressing sensor and lighting conditions adjacent to nuScenes' multimodal dataset focus.

  5. Close Builds on this paper
    Probabilistic 3D Multi-Object Tracking for Autonomous Driving
    Hsu-kuang Chiu, Antonio Prioletti, Jie Li et al. · 2020 · arXiv.org · preprint, not peer reviewed · cited 154 times · Not summarized yet

    Probabilistic 3D Multi-Object Tracking presents a Kalman-filter method that won the nuScenes Tracking Challenge.

    Abstract

    3D multi-object tracking is a key module in autonomous driving applications that provides a reliable dynamic representation of the world to the planning module. In this paper, we present our on-line tracking method, which made the first place in the NuScenes Tracking Challenge, held at the AI Driving Olympics Workshop at NeurIPS 2019. Our method estimates the object states by adopting a Kalman Filter. We initialize the state covariance as well as the process and observation noise covariance with statistics from the training set. We also use the stochastic information from the Kalman Filter in the data association step by measuring the Mahalanobis distance between the predicted object states …

Papers that cite it

  1. Close Same question
    Scalability in Perception for Autonomous Driving: Waymo Open Dataset
    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla et al. · 2019 · Computer Vision and Pattern Recognition · cited 4,635 times · Not summarized yet

    Waymo Open Dataset introduces synchronized, calibrated lidar and camera driving data for research on scalable autonomous-vehicle perception.

    Abstract

    The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the over-all viability of the technology. In an effort to help align the research community’s contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data capt…

  2. Close Broader context Uses this paper in depth
    LynxScenes: A Large-Scale Real-World Benchmark for Outdoor Wheel-Legged Robot Perception, Navigation, Imagination and Reconstruction
    Jun Tian, Hao-Rui Wu, Ruo Chen et al. · 2026 · Not summarized yet

    LynxScenes provides synchronized multimodal data and benchmarks for outdoor wheel-legged robot tasks, rather than autonomous-car perception.

    Abstract

    Wheel-legged robots operating in outdoor spaces shared with humans require accurate perception, socially compliant navigation, future-scene reasoning, and scene reconstruction. However, existing datasets typically support these capabilities in isolation and rarely provide dense spatial and behavior-level supervision on real wheel-legged platforms. We introduce LynxScenes, a large-scale real-world dataset and benchmark collected with a LYNX M20 wheel-legged robot. It contains approximately 100K 30-s navigation clips spanning 833.6 h and 3,294.7 km, with 3.0M annotated frames. The synchronized recordings include four fisheye cameras, dual LiDARs, IMU, odometry, and robot states. Annotations co…

  3. Close Builds on this paper Uses this paper in depth
    PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous Driving
    Yi Feng, Zi-Zhan Guo, E. Junwu et al. · 2026 · Not summarized yet

    PanoBench includes a nuScenes branch with unified interfaces and evaluation protocols for panoptic occupancy prediction.

    Abstract

    Panoptic occupancy unifies 3D geometry, semantics, and instance identities, but its development is limited by scarce instance-aware annotations. Existing benchmarks mainly provide semantic labels reconstructed from incomplete sensor observations, while panoptic datasets have limited domain coverage or annotation completeness. We introduce PanoBench, a camera-centric panoptic occupancy prediction benchmark spanning synthetic and real-world driving scenes. It combines PanoBench-CarlaOcc, PanoBench-nuScenes, and PanoBench-KITTI360 through canonical data interfaces and unified evaluation protocols, providing multi-resolution annotations with persistent instance identities. The synthetic branch d…

  4. Close Review Cited by this paper
    Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
    Di Feng, Christian Haase-Schütz, Lars Rosenbaum et al. · 2019 · IEEE transactions on intelligent transportation systems (Print) · cited 1,423 times · Not summarized yet

    This review surveys multimodal detection and segmentation methods, datasets, and challenges for autonomous-driving perception.

    Abstract

    Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detect…

  5. Close Same method, another problem Uses this paper in depth
    Adversarial Robustness of Camera-Radar Sensor Fusion for Autonomous Driving
    Ce Zhou, Yu-Bo Wu, K. Nguyen et al. · 2026 · Not summarized yet

    This study evaluates camera-radar fusion robustness on nuScenes for 3D detection and depth estimation under simulated attacks.

    Abstract

    Autonomous driving (AD) relies on multi-sensor perception to interpret the surrounding environment. However, the robustness of camera--radar fusion under adversarial conditions remains largely unexplored. This paper presents a systematic measurement study of the robustness of camera--radar sensor fusion in AD. We construct a unified evaluation pipeline on the nuScenes dataset that simulates attacks on both sensing modalities, including eight camera-image perturbations and three radar point-cloud perturbations. We evaluate five camera--radar fusion algorithms across two perception tasks, 3D object detection and depth estimation, and quantify their performance under clean and attacked inputs u…

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 1 November 2026.

The illustrated guide to the papers around a paper

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.