# MERLOT: Multimodal Neural Script Knowledge Models

**Authors:** Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, Yejin Choi  
**arXiv:** [2106.02636](https://arxiv.org/abs/2106.02636)  
**Date:** 2021-06-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.02636v1/

## Key points

- MERLOT is a multimodal neural script knowledge model
- It learns to understand events in the visual world contextually
- Trained in a self-supervised manner by watching YouTube videos with transcribed speech
- Uses frame-level and video-level objectives during pretraining
- Exhibits strong representations of temporal commonsense
- Achieves state-of-the-art performance on 12 different video question-answering datasets when finetuned
- Can transfer well to static images without explicit temporal information
- Achieves an accuracy of 80.6% on Visual Commonsense Reasoning tasks, outperforming similar-sized models by over 3%
- Training on videos rather than static images is crucial for effective learning of multimodal script knowledge
- Scaling up the magnitude and diversity of the pretraining video corpus enhances performance significantly
- Using diverse objectives that encourage full-stack multimodal reasoning contributes to improved results
- MERLOT leverages large-scale unlabeled video data with transcribed speech for learning multimodal script knowledge

## Summary

MERLOT is a multimodal neural script knowledge model that learns to understand events in the visual world contextually by performing multimodal reasoning across time. It is trained in a self-supervised manner by watching millions of YouTube videos with transcribed speech without relying on any labeled data. To learn multimodal script knowledge, MERLOT uses a combination of frame-level (spatial) and video-level (temporal) objectives during pretraining. This enables the model to match images to corresponding words in time and also contextualize global events over time. As a result, MERLOT exhibits strong representations of temporal commonsense and achieves state-of-the-art performance on 12 different video question-answering datasets when finetuned. Additionally, it can transfer well to static images allowing the model to reason about the dynamic context behind visual scenes even without explicit temporal information. On Visual Commonsense Reasoning tasks, MERLOT achieves an impressive accuracy of 80.6%, outperforming similar-sized state-of-the-art models by over 3%. Ablation analyses conducted on MERLOT show that training on videos rather than static images is crucial for learning effective multimodal script knowledge; scaling up the magnitude and diversity of the pretraining video corpus enhances its performance significantly; and using diverse objectives that encourage full-stack multimodal reasoning from recognition to cognition level contributes to improved results. In conclusion, MERLOT presents a powerful approach for learning multimodal script knowledge by leveraging large-scale unlabeled video data with transcribed speech. Its ability to reason across time and transfer knowledge effectively makes it a valuable tool for various visual understanding tasks.