# Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

**Authors:** Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, Karsten Kreis  
**arXiv:** [2304.08818](https://arxiv.org/abs/2304.08818)  
**Date:** 2023-04-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.08818v2/

## Key points

- Authors explore the potential of using Latent Diffusion Models (LDMs) for high-resolution video generation
- LDMs generate high-quality images while minimizing computational demands by training a diffusion model in a compressed lower-dimensional latent space
- Researchers pre-train an LDM on images and extend it into a video generator by incorporating a temporal dimension to the latent space diffusion model and fine-tuning it on encoded image sequences (videos)
- Diffusion model upsamplers are aligned temporally to create consistent video super resolution models
- Study focuses on simulating driving data and creative content creation using text-to-video modeling
- Approach validated on real driving videos with a resolution of 512 x 1024, achieving state-of-the-art performance
- Method can leverage existing pre-trained image LDMs by only requiring training of a temporal alignment model, enabling transformation into an efficient text-to-video model capable of resolutions up to 1280 x 2048
- Temporal layers trained in this manner generalize well to various fine-tuned text-to-image LDMs, allowing for personalized text-to-video generation

## Summary

In their paper "Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models," authors Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis explore the potential of using Latent Diffusion Models (LDMs) for high-resolution video generation. LDMs are known for their ability to generate high-quality images while minimizing computational demands by training a diffusion model in a compressed lower-dimensional latent space. The researchers first pre-train an LDM on images and then extend it into a video generator by incorporating a temporal dimension to the latent space diffusion model and fine-tuning it on encoded image sequences (videos). They also align the diffusion model upsamplers temporally to create consistent video super resolution models. The study focuses on two real-world applications: simulating driving data and creative content creation using text-to-video modeling. The team validates their approach on real driving videos with a resolution of 512 x 1024 and achieves state-of-the-art performance. Importantly, their method can easily leverage existing pre-trained image LDMs by only requiring training of a temporal alignment model in such cases. This allows them to transform the widely available text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model capable of resolutions up to 1280 x 2048. Furthermore, the authors demonstrate that the temporal layers trained in this manner generalize well to various fine-tuned text-to-image LDMs. This property enables them to present initial results for personalized text-to-video generation, opening up exciting possibilities for future content creation endeavors. The research was presented at CVPR 2023 and more information can be found on their project page at https://research.nvidia.com/labs/toronto-ai/VideoLDM/.