# Video Joint Modelling Based on Hierarchical Transformer for Co-summarization

**Authors:** Li Haopeng, Ke Qiuhong, Gong Mingming, Zhang Rui  
**arXiv:** [2112.13478](https://arxiv.org/abs/2112.13478)  
**Date:** 2021-12-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.13478v2/

## Key points

- Video summarization aims to generate a concise summary or storyboard of a video for efficient retrieval and browsing
- Existing methods focus on summarizing individual videos without considering correlations among similar videos
- The authors propose Video Joint Modelling based on Hierarchical Transformer (VJMHT) for co-summarization, which consists of two layers of Transformer models
- VJMHT captures fine-grained details and semantics within each shot of similar videos in the first layer
- The second layer performs shot-level video joint modelling to aggregate cross-video semantic information
- VJMHT explicitly models and learns complete cross-video high-level patterns for summarizing individual videos
- Transformer-based video representation reconstruction is introduced to ensure high-level similarity between the generated summaries and original videos
- VJMHT outperforms existing approaches in terms of F-measure and rank-based evaluation metrics
- It is an effective approach for co-summarization as it considers semantic dependencies across videos while leveraging hierarchical Transformers to capture comprehensive cross-video patterns

## Summary

Video summarization aims to automatically generate a concise summary or storyboard of a video, which is essential for efficient video retrieval and browsing. However, most existing methods focus on summarizing individual videos without considering the correlations among similar videos. These correlations can provide valuable insights for video understanding and summarization. To address this limitation, the authors propose a novel approach called Video Joint Modelling based on Hierarchical Transformer (VJMHT) for co-summarization. VJMHT consists of two layers of Transformer models. In the first layer, semantic representations are extracted from individual shots of similar videos in order to capture the fine-grained details and semantics within each shot. The second layer performs shot-level video joint modelling to aggregate cross-video semantic information. This allows VJMHT to explicitly model and learn complete cross-video high-level patterns that are informative for summarizing individual videos. To ensure that the generated summaries maintain high-level similarity with the original videos, the authors introduce Transformer-based video representation reconstruction. This technique maximizes the similarity between the summary and the original video by reconstructing the video representation using Transformers. The effectiveness of VJMHT is extensively evaluated through experiments comparing it with other state-of-the-art methods in terms of F-measure and rank-based evaluation metrics. The results demonstrate that VJMHT outperforms existing approaches, highlighting its superiority in generating accurate and informative summaries. In conclusion, Video Joint Modelling based on Hierarchical Transformer (VJMHT) is an effective approach for co-summarization of videos as it considers semantic dependencies across videos while leveraging hierarchical Transformers to capture comprehensive cross-video patterns while maintaining high level similarity with original videos. The experimental results validate its effectiveness in generating superior video summaries compared to existing methods.