# Talk2Car: Taking Control of Your Self-Driving Car

**Authors:** Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, Marie-Francine Moens  
**arXiv:** [1909.10838](https://arxiv.org/abs/1909.10838)  
**Date:** 2019-09-24  
**Summary page:** https://www.summarizepaper.com/en/paper/1909.10838v2/

## Key points

- The paper addresses the goal of artificial intelligence to have an agent execute commands communicated through natural language in a visual environment shared by humans and the agent.
- The problem is specifically considered in an autonomous driving setting where a passenger requests an action associated with an object found in a street scene.
- The authors present the Talk2Car dataset, which is the first object referral dataset containing commands written in natural language for self-driving cars.
- The dataset is compared with related datasets such as ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscape-Ref, and CLEVR-Ref.
- Strong state-of-the-art models are used to analyze their performance on the proposed object referral task.
- Promising results are achieved by these models, but further research is still required in natural language processing and computer vision to improve their performance.
- The Talk2Car dataset can be accessed on the authors' website.
- The paper is authored by Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie-Francine Moens from KU Leuven's Department of Computer Science (CS) and Department of Electrical Engineering (ESAT).

## Summary

The paper titled "Talk2Car: Taking Control of Your Self-Driving Car" addresses the long-term goal of artificial intelligence to have an agent execute commands communicated through natural language that are grounded in a visual environment shared by the human giving the command and the agent. Specifically, this problem is considered in an autonomous driving setting where a passenger requests an action associated with an object found in a street scene. To facilitate research in this area, the authors present the Talk2Car dataset which is the first object referral dataset containing commands written in natural language for self-driving cars. The dataset is compared with related datasets such as ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscape-Ref and CLEVR-Ref. Strong state-of-the-art models are used to analyze their performance on the proposed object referral task. The results show that while promising results are achieved by these models further research is still required in natural language processing and computer vision to improve their performance. The Talk2Car dataset can be accessed on the authors' website. The paper is authored by Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool and Marie-Francine Moens from KU Leuven's Department of Computer Science (CS) and Department of Electrical Engineering (ESAT).