# Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

**Authors:** Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, Joshua B. Tenenbaum  
**arXiv:** [1810.02338](https://arxiv.org/abs/1810.02338)  
**Date:** 2018-10-04  
**Summary page:** https://www.summarizepaper.com/en/paper/1810.02338v1/

## Key points

- The paper proposes a novel approach to visual question answering (VQA) by combining deep representation learning for visual recognition and language understanding with symbolic program execution for reasoning.
- The proposed neural-symbolic VQA system recovers a structural scene representation from the image and a program trace from the question, then executes the program on the scene representation to obtain an answer.
- Incorporating symbolic structure as prior knowledge is more robust to long program traces, allowing the model to solve complex reasoning tasks better.
- On the CLEVR dataset, the proposed model achieved an accuracy of 99.8%.
- This approach is more data- and memory-efficient compared to existing methods for offline question answering since it performs well after learning on a small number of training data and can encode an image into a compact representation requiring less storage.
- It offers full transparency to the reasoning process, enabling researchers to interpret and diagnose each execution step.
- This level of transparency allows for better understanding of how different components contribute to overall performance.
- Overall, this paper presents a promising new approach to VQA that disentangles reasoning from vision and language understanding through neural-symbolic integration.

## Summary

The paper titled "Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding" proposes a novel approach to visual question answering (VQA) by combining deep representation learning for visual recognition and language understanding with symbolic program execution for reasoning. The proposed neural-symbolic VQA system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. One of the unique advantages of incorporating symbolic structure as prior knowledge is that executing programs on a symbolic space is more robust to long program traces, allowing the model to solve complex reasoning tasks better. In fact, on the CLEVR dataset, the proposed model achieved an accuracy of 99.8%. Additionally, this approach is more data- and memory-efficient compared to existing methods for offline question answering since it performs well after learning on a small number of training data and can encode an image into a compact representation requiring less storage. Another significant advantage is that it offers full transparency to the reasoning process, enabling researchers to interpret and diagnose each execution step. This level of transparency allows for better understanding of how different components contribute to overall performance. Overall, this paper presents a promising new approach to VQA that disentangles reasoning from vision and language understanding through neural-symbolic integration. The proposed model achieves state-of-the-art results while also offering improved efficiency and transparency in its reasoning process, making it an attractive solution for complex visual question answering tasks.