# HawkEye: Training Video-Text LLMs for Grounding Text in Videos

**Authors:** Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, Dongyan Zhao  
**arXiv:** [2403.10228](https://arxiv.org/abs/2403.10228)  
**Date:** 2024-03-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.10228v1/

## Key points

- Video-text Large Language Models (LLMs) struggle with understanding and grounding text queries in longer and more complex videos
- Long-form videos like movies, tutorials, and documentaries are crucial for conveying information, knowledge, opinions, and emotions
- The paper focuses on enhancing the temporal video grounding abilities of existing video-text LLMs
- HawkEye is a proposed solution that improves multi-modal understanding abilities through targeted training on long-form videos
- HawkEye demonstrates superior performance in various video-text tasks such as temporal video grounding, question grounding, and video question answering

## Summary

, , , , 


In the realm of video-text Large Language Models (LLMs), while these models have shown impressive capabilities in answering questions and engaging in conversations based on simple videos, they struggle when it comes to understanding and grounding text queries in longer and more complex videos. This is a significant limitation as long-form videos such as movies, tutorials, and documentaries play a crucial role in conveying a wealth of information, knowledge, opinions, and emotions. To address this challenge, this paper focuses on enhancing the temporal video grounding abilities of existing video-text LLMs. <kw>Video-text LLMs:</kw> In recent years, video-text LLMs have gained attention for their ability to answer questions and engage in conversations based on simple videos. However, they face challenges when it comes to understanding longer and more complex videos. <kw>Temporal video grounding:</kw> The main focus of this paper is to improve the temporal video grounding abilities of existing video-text LLMs. This task involves identifying specific segments related to given text queries within a longer video containing multiple actions and events. <kw>Long-form videos:</kw> Long-form videos such as movies, tutorials, and documentaries are essential for conveying a wide range of information, knowledge, opinions, and emotions. However, current video-text LLMs struggle with understanding these types of videos. <kw>HawkEye:</kw> The proposed solution involves targeted training on long-form videos without extensively modifying pre-training or visual-text alignment stages. This results in HawkEye - one of the first fully text-to-text capable video-text LLMs with advanced multi-modal understanding abilities. <kw>Multi-modal understanding abilities:</kw> Through effective representation formats and a large-scale time-aware training dataset, HawkEye showcases its superior performance in various video-text tasks, including temporal video grounding, question grounding, and video question answering.