# VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding

**Authors:** Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, Qing Li  
**arXiv:** [2403.11481](https://arxiv.org/abs/2403.11481)  
**Date:** 2024-03-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.11481v1/

## Key points

- Researchers propose a novel multimodal agent called VideoAgent to address video understanding challenges
- VideoAgent reconciles large language models (LLMs) and vision-language models with a unified memory mechanism
- Focus is on capturing long-term temporal relations in lengthy videos
- VideoAgent utilizes structured memory for storing temporal event descriptions and object-centric tracking states of the video
- Results show impressive performances on long-horizon video understanding benchmarks, with significant improvements over baselines
- Study discusses challenges faced by existing models in processing long-form videos and explores solutions like spatial and temporal sampler modules and scaling multimodal models to longer videos
- Research has focused on augmenting LLMs with tools for solving multimodal tasks without extensive training

## Summary

, , , , 


In this study, the researchers propose a novel multimodal agent called VideoAgent that aims to address the challenging video understanding problem by reconciling large language models (LLMs) and vision-language models with a unified memory mechanism. The key focus is on capturing long-term temporal relations in lengthy videos. <break>
<break>

<break>
<break>

VideoAgent utilizes a structured memory to store both generic temporal event descriptions and object-centric tracking states of the video. When given an input task query, VideoAgent employs tools such as video segment localization and object memory querying, along with other visual foundation models, to interactively solve the task. Notably, VideoAgent leverages the zero-shot tool-use ability of LLMs. The results demonstrate impressive performances on several long-horizon video understanding benchmarks, showcasing an average increase of 6.6% on NExT-QA and 26.0% on EgoSchema over baselines. This performance improvement closes the gap between open-sourced models and private counterparts like Gemini 1.5 Pro. Furthermore, the study discusses related work in the field of multimodal LLMs for video understanding, highlighting challenges faced by existing models in processing long-form videos. To address these challenges, approaches such as utilizing spatial and temporal sampler modules for extracting optical flow-based temporal features (LSTP) and scaling multimodal models to longer videos with massive private datasets (Gemini) have been explored. Additionally, research has focused on augmenting LLMs with tools to solve multimodal tasks without extensive training. Examples include VisProg, which equips GPT-3 with visual tools for solving complex visual reasoning problems. <break>
<break>

Overall, this study contributes to advancing video understanding through the development of a sophisticated multimodal agent that effectively integrates different foundation models and memory mechanisms to tackle long-term temporal relations in videos.