# World Model on Million-Length Video And Language With Blockwise RingAttention

**Authors:** Hao Liu, Wilson Yan, Matei Zaharia, Pieter Abbeel  
**arXiv:** [2402.08268](https://arxiv.org/abs/2402.08268)  
**Date:** 2024-02-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.08268v2/

## Key points

- Authors address limitations of current language models in understanding complex aspects of the world
- Value of video sequences in providing temporal information for joint modeling with language
- Challenges of learning from millions of tokens within video and language sequences
- Curating a large dataset comprising diverse videos and books to overcome obstacles
- Employing Blockwise RingAttention technique to efficiently train on long sequences
- Training one of the largest context size transformers on long video and language sequences
- Proposing solutions for vision-language training challenges, including masked sequence packing and loss weighting
- Presenting a highly optimized implementation featuring RingAttention, Blockwise Transformers, and other essential features
- Open-sourcing a family of 7B parameter models capable of processing extensive text documents and videos containing over 1M tokens
- Paving the way for training on massive datasets encompassing long video and language sequences to enhance AI capabilities

## Summary

In their paper titled "World Model on Million-Length Video And Language With Blockwise RingAttention," authors Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel address the limitations of current language models in understanding complex aspects of the world that are not easily described in words. They highlight the value of video sequences in providing temporal information that is absent in static images and language, making them an attractive option for joint modeling with language. By combining video and language data, these models have the potential to develop a comprehensive understanding of both human textual knowledge and the physical world, thereby enhancing AI capabilities for assisting humans. However, learning from millions of tokens within video and language sequences presents challenges such as memory constraints, computational complexity, and limited datasets. To overcome these obstacles, the authors curate a large dataset comprising diverse videos and books. They also employ the Blockwise RingAttention technique to efficiently train on long sequences while gradually increasing context size from 4K to 1M tokens. The key contributions of this work include training one of the largest context size transformers on long video and language sequences, which sets new benchmarks in challenging retrieval tasks and enhances comprehension of lengthy videos. The authors propose solutions for overcoming vision-language training challenges by utilizing masked sequence packing to mix different sequence lengths, implementing loss weighting to balance language and vision components, and creating a model-generated QA dataset for facilitating long sequence chat. Furthermore, they present a highly optimized implementation featuring RingAttention, Blockwise Transformers, masked sequence packing,
and other essential features for training on multimodal sequences with millions-length tokens. The authors have fully open-sourced a family of 7B parameter models capable of processing extensive text documents (LWM-Text,LWM-Text-Chat) as well as videos (LWM,LWM-Chat) containing over 1M tokens. Overall, this research paves the way for training on massive datasets encompassing long video and language sequences to enhance understanding across various domains including human knowledge and the multimodal world. It represents a significant step towards developing broader AI capabilities that can assist humans more effectively in diverse tasks.