# MultiDiff: Consistent Novel View Synthesis from a Single Image

**Authors:** Norman Müller, Katja Schwarz, Barbara Roessle, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, Peter Kontschieder  
**arXiv:** [2406.18524](https://arxiv.org/abs/2406.18524)  
**Date:** 2024-06-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.18524v1/

## Key points

- MultiDiff is a novel approach for consistent novel view synthesis from a single RGB image
- Incorporates strong priors such as monocular depth predictors and video-diffusion models to enhance geometric stability in target views
- Utilizes a structured noise distribution to further enhance consistency and image quality
- Simultaneously synthesizes a sequence of frames, resulting in high-quality and multi-view consistent results even for long-term scene generation with large camera movements
- Outperforms state-of-the-art methods on challenging real-world datasets like RealEstate10K and ScanNet
- Evaluation metrics include Peak Signal-to-Noise Ratio (PSNR), perceptual similarity (LPIPS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), Fréchet Video Distance (FVD), and symmetric epipolar distance (SED)
- Excels in both short-term and long-term view synthesis scenarios, achieving superior FID and KID scores on datasets like RealEstate10K and ScanNet at different resolutions

## Summary

, , , , 


In this work, we introduce MultiDiff, a novel approach for consistent novel view synthesis of scenes from a single RGB image. The task of synthesizing novel views from a single reference image is inherently challenging due to the presence of multiple plausible explanations for unobserved areas. To address this issue, we incorporate strong priors in the form of monocular depth predictors and video-diffusion models. By leveraging monocular depth, our model can be conditioned on warped reference images for target views, enhancing geometric stability. The video-diffusion prior provides a robust proxy for 3D scenes, enabling the model to learn continuous and pixel-accurate correspondences across generated images. A novel approach for consistent novel view synthesis from a single RGB image. Challenging task with multiple plausible explanations for unobserved areas. Strong prior used to enhance geometric stability in target views. Robust proxy for 3D scenes that enables continuous and pixel-accurate correspondences in generated images. Ability to edit multiple views without additional tuning. Unlike approaches relying on autoregressive image generation that are susceptible to drifts and error accumulation, MultiDiff simultaneously synthesizes a sequence of frames, resulting in high-quality and multi-view consistent results even for long-term scene generation with large camera movements. Additionally, our approach significantly reduces inference time by an order of magnitude. To further enhance consistency and image quality, we introduce a novel structured noise distribution. Experimental results demonstrate that MultiDiff outperforms state-of-the-art methods on challenging real-world datasets such as RealEstate10K and ScanNet. Our model also supports multi-view consistent editing without the need for additional tuning. In terms of evaluation metrics, we assess the performance of our approach using various measures including Peak Signal-to-Noise Ratio (PSNR), perceptual similarity (LPIPS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), Fréchet Video Distance (FVD), and symmetric epipolar distance (SED). We compare our method against existing techniques such as DFM and PhotoNVS on tasks like short-term and long-term view synthesis. Furthermore, we highlight the efficiency of MultiDiff in generating multiple frames from a single input image in parallel compared to other methods like PhotoNVS and DFM. Challenging real-world dataset used for evaluation. Another challenging real-world dataset used for evaluation. Evaluation metric used to measure image quality. Evaluation metric used to measure image similarity. Evaluation metric used to measure distribution similarity between generated images and ground truth images. Similar to FID, but uses kernel density estimation for improved accuracy. Evaluation metric used to measure video quality by comparing distributions of generated videos with ground truth videos. Evaluation metric used to measure geometric consistency between generated views. Our approach excels in both short-term and long-term view synthesis scenarios, achieving superior FID and KID scores on datasets like RealEstate10K and ScanNet at different resolutions.