# Language Prompt for Autonomous Driving

**Authors:** Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, Jianbing Shen  
**arXiv:** [2309.04379](https://arxiv.org/abs/2309.04379)  
**Date:** 2023-09-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.04379v1/

## Key points

- Growing trend in computer vision community to capture objects based on natural language prompts
- Lack of paired prompt-instance data for driving scenarios
- Introduction of NuPrompt - object-centric language prompt set for driving scenes in 3D, multi-view, and multi-frame space
- NuPrompt consists of 35,367 language descriptions referring to an average of 5.3 object tracks
- Introduction of PromptTrack - a baseline model based on Transformer architecture
- Experimental results show impressive performance of PromptTrack on NuPrompt
- Dataset and code will be made publicly available at GitHub repository (https://github.com/wudongming97/Prompt4Driving)
- Research addresses bottleneck in utilizing language prompts for driving scenarios
- Promising results from PromptTrack contribute to advancing research in autonomous driving.

## Summary

In the computer vision community, there is a growing trend to capture objects of interest based on natural language prompts given by humans. To overcome the challenge of lack of paired prompt-instance data for driving scenarios, the authors propose NuPrompt - the first object-centric language prompt set specifically designed for driving scenes in 3D, multi-view and multi-frame space. NuPrompt consists of 35,367 language descriptions which refer to an average of 5.3 object tracks providing a rich source of information for training and evaluation purposes. The authors introduce PromptTrack - a simple end-to-end baseline model based on Transformer architecture - to demonstrate the effectiveness of their approach. Experimental results show that PromptTrack achieves impressive performance on NuPrompt. The authors plan to make both the dataset and code publicly available at their GitHub repository (https://github.com/wudongming97/Prompt4Driving). This research addresses an important bottleneck in utilizing language prompts for driving scenarios by introducing NuPrompt and formulating a new prompt-based driving task with promising results from PromptTrack model contributing to advancing research in autonomous driving.