# TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models

**Authors:** Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, Arman Cohan  
**arXiv:** [2410.23266](https://arxiv.org/abs/2410.23266)  
**Date:** 2024-10-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.23266v1/

## Key points

- Multimodal Foundation Models (MFMs) praised for leveraging temporal context in video understanding
- Existing benchmarks suggest MFMs may lack visual temporal reasoning capabilities
- New benchmark TOMATO introduced to address discrepancy, consisting of 1,484 questions across six tasks applied to 1,417 videos
- TOMATO includes self-recorded human-centric scenarios, interactive gestures, and simulated scenarios using Keynote and 3D modeling frameworks
- Annotation process focused on crafting questions requiring reasoning across all frames for rigorous evaluation of temporal reasoning abilities
- Significant performance gap between human and model performance in TOMATO analysis
- Current MFMs struggle to interpret frames as a continuous sequence and rely heavily on common sense rather than true visual reasoning
- Models often incorrectly interpret visual cues or fail to infer sequential patterns accurately
- TOMATO serves as a critical testbed for evaluating next generation of MFMs and emphasizes need for AI systems capable of comprehending human world dynamics through video modalities

## Summary

Multimodal Foundation Models (MFMs) have been highly praised for their exceptional performance in leveraging temporal context in the realm of video understanding. However, a closer examination of existing benchmarks reveals that these models may not possess the visual temporal reasoning capabilities they are credited with. Many tasks can be solved using only a few frames or frames out of order, indicating a potential overestimation of MFMs' abilities. To address this discrepancy, a new benchmark called TOMATO (Temporal Reasoning Multimodal Evaluation) has been introduced. TOMATO consists of 1,484 meticulously curated questions across six tasks applied to 1,417 videos. These videos include self-recorded and -generated human-centric scenarios as well as additional recordings featuring interactive gestures and simulated scenarios using Keynote and 3D modeling frameworks. The annotation process focused on crafting questions that require reasoning across all frames to ensure a more rigorous evaluation of temporal reasoning abilities. A meticulous quality check process was implemented to maintain consistency and accuracy in the annotated question-answer pairs. The analysis of TOMATO reveals a significant gap between human and model performance, highlighting fundamental limitations in current MFMs' ability to interpret frames as a continuous sequence. It is evident that models often struggle to reason across multiple time steps and rely heavily on common sense rather than true visual reasoning. This is further supported by instances where models incorrectly interpret visual cues or fail to infer sequential patterns accurately. Overall, TOMATO serves as a critical testbed for evaluating the next generation of MFMs and emphasizes the need for AI systems capable of comprehending human world dynamics through video modalities. The benchmark challenges existing models to improve their temporal reasoning capabilities and move beyond simple recognition tasks towards a deeper understanding of dynamic visual sequences.