# Learning and Verification of Task Structure in Instructional Videos

**Authors:** Medhini Narasimhan, Licheng Yu, Sean Bell, Ning Zhang, Trevor Darrell  
**arXiv:** [2303.13519](https://arxiv.org/abs/2303.13519)  
**Date:** 2023-03-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.13519v1/

## Key points

- Abundance of instructional videos available online makes it possible to learn a diverse range of multi-step task models from these videos
- VideoTaskformer is a new pre-trained video model that focuses on representing the semantics and structure of instructional videos
- Model is pre-trained using a simple yet effective objective - predicting weakly supervised textual labels for steps that are randomly masked out from an instructional video (masked step modeling)
- VideoTaskformer involves learning step representations globally by leveraging the entire surrounding task as context
- Using learned representations, authors can verify if an unseen video correctly executes a given task and forecast which steps are likely to be taken after a given step
- Two new benchmarks introduced for detecting mistakes in instructional videos - verifying if there is an anomalous step and ensuring that steps are executed in the correct order
- Long-term forecasting benchmark introduced where goal is to predict long-range future steps from a given step
- Method outperforms previous baselines on these tasks, demonstrating its effectiveness in measuring quality of step representations
- VideoTaskformer evaluated on three existing benchmarks - procedural activity recognition, step classification, and step forecasting - and demonstrates approach outperforms existing baselines while achieving new state-of-the-art performance
- Unsupervised pre-training using neural networks with automatic speech recognition (ASR) outperforms previous unsupervised methods
- Competitive linear-probe performance reported and improved results when adding task labels
- Results from evaluating approach on activity recognition in EPIC Kitchens-100 included
- Model's performance on the step localization task in COIN reported

## Summary

The abundance of instructional videos available online has made it an attractive goal to learn a diverse range of multi-step task models from these videos. To achieve this, the authors introduce a new pre-trained video model called VideoTaskformer that focuses on representing the semantics and structure of instructional videos. The model is pre-trained using a simple yet effective objective - predicting weakly supervised textual labels for steps that are randomly masked out from an instructional video (masked step modeling). Unlike prior work that learns step representations locally, VideoTaskformer involves learning them globally by leveraging the entire surrounding task as context. Using the learned representations, the authors can verify if an unseen video correctly executes a given task and forecast which steps are likely to be taken after a given step. They also introduce two new benchmarks for detecting mistakes in instructional videos - verifying if there is an anomalous step and ensuring that steps are executed in the correct order. Additionally, they introduce a long-term forecasting benchmark where the goal is to predict long-range future steps from a given step. The authors' method outperforms previous baselines on these tasks, demonstrating its effectiveness in measuring the quality of step representations. Furthermore, they evaluate VideoTaskformer on three existing benchmarks - procedural activity recognition, step classification, and step forecasting - and demonstrate that their approach outperforms existing baselines while achieving new state-of-the-art performance. In addition to their evaluation results, the authors note that their unsupervised pre-training using neural networks with automatic speech recognition (ASR) outperforms previous unsupervised methods. They also report competitive linear-probe performance and improved results when adding task labels. Finally, they include results from evaluating their approach on activity recognition in EPIC Kitchens-100 and report their model's performance on the step localization task in COIN. Overall, this method opens up possibilities for learning how to execute various tasks by watching instructional videos such as cooking complicated meals by watching cooking shows.