# Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

**Authors:** Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, Karsten Kreis  
**arXiv:** [2304.08818](https://arxiv.org/abs/2304.08818)  
**Date:** 2023-04-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.08818v1/

## Key points

- The paper presents a novel approach to high-resolution video generation using Latent Diffusion Models (LDMs).
- LDMs enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion model in a compressed lower-dimensional latent space.
- The proposed method involves pre-training an LDM on images only and then introducing a temporal dimension to the latent space diffusion model to turn the image generator into a video generator.
- The authors fine-tune the model on encoded image sequences or videos and temporally align diffusion model upsamplers to turn them into temporally consistent video super resolution models.
- The authors focus on two real-world applications: simulation of in-the-wild driving data and creative content creation with text-to-video modeling.
- They validate their Video LDM on real driving videos of resolution 512 x 1024 and achieve state-of-the art performance.
- Their approach can easily leverage off the shelf pre trained image LDMs as they only need to train a temporal alignment model in that case.
- By doing so they turn the publicly available state of the art text to image LDM Stable Diffusion into an efficient and expressive text to video model with resolution up to 1280 x 2048.
- The authors show that the temporal layers trained in this way generalize to different fine tuned text to image LDMs, enabling personalized text-to-video generation for creative content creation.
- Overall, the proposed method enables high quality video synthesis while avoiding excessive compute demands.

## Summary

The paper "Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models" by Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler and Karsten Kreis presents a novel approach to high-resolution video generation using Latent Diffusion Models (LDMs). LDMs enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion model in a compressed lower-dimensional latent space. The authors extend this paradigm to the resource-intensive task of high-resolution video generation. The proposed method involves pre-training an LDM on images only and then introducing a temporal dimension to the latent space diffusion model to turn the image generator into a video generator. The authors fine-tune the model on encoded image sequences or videos and temporally align diffusion model upsamplers to turn them into temporally consistent video super resolution models. The authors focus on two real-world applications: simulation of in-the-wild driving data and creative content creation with text-to-video modeling. They validate their Video LDM on real driving videos of resolution 512 x 1024 and achieve state-of-the art performance. Furthermore, their approach can easily leverage off the shelf pre trained image LDMs as they only need to train a temporal alignment model in that case. By doing so they turn the publicly available state of the art text to image LDM Stable Diffusion into an efficient and expressive text to video model with resolution up to 1280 x 2048. The authors show that the temporal layers trained in this way generalize to different fine tuned text to image LDMs. Utilizing this property they demonstrate personalized text to video generation for creative content creation which opens exciting directions for future content creation. Overall, the proposed method enables high quality video synthesis while avoiding excessive compute demands. It achieves state of the art performance on real driving videos and demonstrates promising results for personalized text to video generation. The authors provide a project page with more information: https://research.nvidia.com/labs/toronto -ai/VideoLDM/.