# Attention is all you need for Videos: Self-attention based Video Summarization using Universal Transformers

**Authors:** Manjot Bilkhu, Siyang Wang, Tushar Dobhal  
**arXiv:** [1906.02792](https://arxiv.org/abs/1906.02792)  
**Date:** 2019-06-06  
**Summary page:** https://www.summarizepaper.com/en/paper/1906.02792v1/

## Key points

- Video Captioning and Summarization have gained popularity due to advancements in Sequence Modeling techniques
- Long-Short Term Memory networks (LSTMs) and Gated Recurrent Units (GRUs) are commonly used for modeling dependencies
- Convolutional Neural Networks (CNNs) are used for extracting spatio-temporal features
- Soft attention layers are incorporated with GRUs or LSTMs to improve performance
- Transformer model has revolutionized Sequence Modeling by incorporating self-attention mechanisms
- This project focuses on implementing a Transformer-based model with 3D CNN architectures for video feature extraction
- Dimensionality reduction techniques are applied to manage the overall size of the model
- Experiments on MSVD and ActivityNet datasets show effectiveness in Single and Dense video captioning scenarios.

## Summary

Video Captioning and Summarization have gained significant popularity in recent years due to advancements in Sequence Modeling techniques. Long-Short Term Memory networks (LSTMs) and Gated Recurrent Units (GRUs) are commonly used for modeling dependencies in these tasks. To extract spatio-temporal features, existing architectures employ Convolutional Neural Networks (CNNs) and incorporate soft attention layers with either GRUs or LSTMs. While attention layers improve the performance of recurrent units, they still suffer from inherent drawbacks. The introduction of the Transformer model has revolutionized the field of Sequence Modeling by incorporating self-attention mechanisms. This project focuses on implementing a Transformer-based model that leverages 3D CNN architectures for video feature extraction. Dimensionality reduction techniques are applied to ensure that the overall size of the model remains manageable. Experiments conducted on MSVD and ActivityNet datasets demonstrate the effectiveness of this approach in both Single and Dense video captioning scenarios.