# Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering

**Authors:** Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, Alan Yuille  
**arXiv:** [2406.00622](https://arxiv.org/abs/2406.00622)  
**Date:** 2024-06-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.00622v1/

## Key points

- Understanding dynamic properties of objects and their interactions in 3D scenes from video is crucial for effective reasoning in vision-language models (VLMs).
- The study introduces SuperCLEVR-Physics, a video question answering dataset focusing on dynamic properties like velocity, acceleration, and collisions within 4D scenes.
- Current VLMs struggle with understanding dynamic properties due to a lack of explicit knowledge about spatial structure in 3D and world dynamics across time variants.
- NS-4Dynamics is proposed as a Neural-Symbolic model designed for reasoning on 4D Dynamics properties under explicit scene representation from videos.
- NS-4Dynamics outperforms previous VLMs in understanding dynamic properties, future prediction, factual reasoning, and counterfactual reasoning.
- Visual Question Answering (VQA) plays a vital role in assessing machine learning models' ability to identify objects accurately, understand relationships effectively, and engage in sophisticated reasoning over complex scenes.
- Models dealing with VideoQA scenarios involving dynamic scenes should incorporate a dynamic understanding module for fluid reasoning about object interactions.

## Summary

In the realm of vision-language models (VLMs), grasping the dynamic properties of objects and their interactions within 3D scenes from video is paramount for effective reasoning. This study introduces a novel video question answering dataset, SuperCLEVR-Physics, which specifically hones in on the dynamics properties of objects. The focus lies on physical concepts such as velocity, acceleration, and collisions within 4D scenes, where the model must fully comprehend these dynamic properties to answer questions based on them. Through evaluating various current VLMs, it becomes apparent that these models struggle with understanding dynamic properties due to a lack of explicit knowledge about spatial structure in 3D and world dynamics across time variants. To underscore the significance of an explicit 4D dynamics representation for comprehending world dynamics, the researchers propose NS-4Dynamics – a Neural-Symbolic model designed for reasoning on 4D Dynamics properties under explicit scene representation from videos. By leveraging scene rendering likelihood combined with physical prior distribution, the 4D scene parser can estimate object dynamics over time and interpret observations into 4D scene representations as world states. Through incorporating neural-symbolic reasoning, this approach facilitates advanced applications in future prediction, factual reasoning, and counterfactual reasoning. Experimental results showcase that NS-4Dynamics outperforms previous VLMs in understanding dynamic properties and answering questions related to factual queries, future prediction, and counterfactual reasoning. Furthermore, based on the explicit 4D scene representation provided by this model, it excels in reconstructing 4D scenes and simulating future or counterfactual events with precision. Moreover, delving deeper into visual question answering (VQA) reveals its role as a comprehensive method for assessing machine learning models' ability to identify objects accurately, understand their relationships effectively, and engage in sophisticated reasoning over complex scenes. In VideoQA scenarios involving dynamic scenes like those explored in this study's dataset SuperCLEVR-Physics, it is imperative for models to incorporate a dynamic understanding module that can reason about object interactions fluidly.