Search Results for « Video Captioning »

Abstract Traditional video captioning methods mainly rely on the detection of objects in the video, ignoring the discussion of the mutual semantic relations between objects, resulting in a decrease in description accuracy under the influence of occlusion and tiny-size objects. In this paper, we discuss the hippocampal circuit and memory mechanism of the human brain, and a brain-inspired model based on the long-term short-term memory branch of the hippocampus is proposed. The explicit and implicit semantics in video are explored by simulating the interaction mechanism between short-term memory in the hippocampus and long-term memory in the cerebral cortex. We evaluate the effectiveness of our method on public benchmarks (Microsoft Research Video Description Corpus(MSVD) and Microsoft Research Video to Text(MSRVTT)). Extensive experiments demonstrate that our model outperforms general state-of-the-art methods and effectively improves the effect of content understanding in videos.

Abstract Many video platforms rely on the descriptions that uploaders provide for video retrieval. However, this reliance may cause inaccuracies. Although deep learning-based video captioning can resolve this problem, it has some limitations: (1) traditional keyframe extraction techniques do not consider video length/content, resulting in low accuracy, high storage requirements, and long processing times; (2) Arabic language support in video captioning is not extensive. This study proposes a new video captioning approach that uses an efficient keyframe extraction method and supports both Arabic and English. The proposed keyframe extraction technique uses time- and content-based approaches for better quality captions, fewer storage space requirements, and faster processing. The English and Arabic models use a sequence-to-sequence framework with long short-term memory in both the encoder and decoder. Both models were evaluated on caption quality using four metrics: bilingual evaluation understudy (BLEU), metric for evaluation of translation with explicit ORdering (METEOR), recall-oriented understudy of gisting evaluation (ROUGE-L), and consensus-based image description evaluation (CIDE-r). They were also evaluated using cosine similarity to determine their suitability for video retrieval. The results demonstrated that the English model performed better with regards to caption quality and video retrieval. In terms of BLEU, METEOR, ROUGE-L, and CIDE-r, the English model scored 47.18, 30.46, 62.07, and 59.98, respectively, whereas the Arabic model scored 21.65, 36.30, 44.897, and 45.52, respectively. According to the video retrieval, the English and Arabic models successfully retrieved 67% and 40% of the videos, respectively, with 20% similarity. These models have potential applications in storytelling, sports commentaries, and video surveillance.

Discrete video tokenization has become a central design choice in recent multimodal large language models, shifting visual understanding from continuous patch features toward compact sequences of code indices. While rapid progress has been demonstrated on generative benchmarks, the impact of tokenizer choice on discriminative downstream tasks such as video question answering and video captioning remains under-quantified. This paper presents a controlled empirical comparison of nine discrete video tokenization schemes — VQ-VAE, VQ-GAN, VideoGPT, MAGVIT, MAGVIT-v2, OmniTokenizer, LARP, TiTok, and Cosmos Tokenizer — evaluated under a shared decoder-only language backbone on five open-ended video question answering benchmarks (MSRVTT-QA, MSVD-QA, ActivityNet-QA, NExT-QA, TGIF-QA) and two captioning benchmarks (MSR-VTT, VATEX). Across 63 tokenizer–dataset pairs, modern lookup-free and finite-scalar quantization schemes outperform the classic VQ-VAE baseline by 7.6 to 9.5 accuracy points on short-video question answering and 10.8 to 12.3 CIDEr points on captioning, while holistic query-based tokens deliver the largest gains on causal and temporal reasoning. A factor analysis separates codebook size from spatial-temporal compression, revealing a moderate sweet spot around 65K codebook entries at an 8×8×4 compression ratio, beyond which downstream accuracy plateaus or regresses. The results supply practical guidance for tokenizer selection in video–language pipelines.

Abstract Dense video captioning is a machine translation task that aims to localize events from full video and describe them separately. Human observers who parse the video are also impressed by frames containing a high amount of information and duplication, but video often appears unrelated frames in the subject. However, existing works have largely ignored these details. To fully incorporate human visual perception into the process of understanding video, we propose an enhanced visual multi-modal fusion framework (Evmff), which utilizes the captions of video keyframes to improve dense video captioning performance. We first extract video keyframes through time stamps and then apply the recently proposed image captioning method DLCT to obtain a temporally aligned caption of the keyframe. Evmff fuses the textual information of speech, the caption of image keyframes, video features, and audio features, applying transformer architecture to convert the data into text descriptions. The performance of our model is verified using the ActivityNet Captions dataset with four different indicators, Bleu@N, METEOR, Rouge_L, and CIDEr-D. Ablation experiments show that employing the video keyframe description as the input of the multi-modal model compensates for the deficiency in visual information understanding. Our code will be released.

Generating accurate and contextually rich captions for images and videos is essential for various applications, from assistive technology to content recommendation. However, challenges such as maintaining temporal coherence in videos, reducing noise in large-scale datasets, and enabling real-time captioning remain significant. We introduce MIRA-CAP (Memory-Integrated Retrieval-Augmented Captioning), a novel framework designed to address these issues through three core innovations: a cross-modal memory bank, adaptive dataset pruning, and a streaming decoder. The cross-modal memory bank retrieves relevant context from prior frames, enhancing temporal consistency and narrative flow. The adaptive pruning mechanism filters noisy data, which improves alignment and generalization. The streaming decoder allows for real-time captioning by generating captions incrementally, without requiring access to the full video sequence. Evaluated across standard datasets like MS COCO, YouCook2, ActivityNet, and Flickr30k, MIRA-CAP achieves state-of-the-art results, with high scores on CIDEr, SPICE, and Polos metrics, underscoring its alignment with human judgment and its effectiveness in handling complex visual and temporal structures. This work demonstrates that MIRA-CAP offers a robust, scalable solution for both static and dynamic captioning tasks, advancing the capabilities of vision–language models in real-world applications.