# Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction

**Authors:** Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, Ying-Cong Chen  
**arXiv:** [2409.18124](https://arxiv.org/abs/2409.18124)  
**Date:** 2024-09-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.18124v3/

## Key points

- Authors explore leveraging visual priors from pre-trained text-to-image diffusion models for zero-shot generalization in dense prediction tasks
- Existing methods often adopt original diffusion formulation without considering unique requirements of dense prediction
- Comprehensive analysis of diffusion formulation tailored for dense prediction tasks to enhance quality and efficiency
- Introduction of Lotus, a novel diffusion-based visual foundation model optimized for dense prediction
- Lotus trained to directly predict annotations instead of noise, improving performance without increasing training data or model capacity significantly
- Streamlining the diffusion process into a single-step procedure to simplify optimization and improve inference speed
- Achieves state-of-the-art performance in zero-shot depth and normal estimation across various datasets
- Outperforms existing diffusion-based methods in terms of speed, enabling practical applications such as joint estimation and 3D reconstruction

## Summary

In their paper titled "Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction," authors Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying-Cong Chen explore the potential of leveraging visual priors from pre-trained text-to-image diffusion models to improve zero-shot generalization in dense prediction tasks. The study highlights the limitations of existing methods that often adopt the original diffusion formulation without considering the unique requirements of dense prediction compared to image generation. The authors conduct a comprehensive analysis of the diffusion formulation specifically tailored for dense prediction tasks, focusing on enhancing both quality and efficiency. They identify that the original parameterization type designed for image generation - which involves predicting noise - is not conducive to dense prediction and can introduce harmful variance. Additionally, they note that the multi-step noising/denoising diffusion process is unnecessary and poses challenges in optimization. To address these issues, the authors introduce Lotus, a novel diffusion-based visual foundation model with an innovative adaptation protocol optimized for dense prediction. Unlike traditional approaches, Lotus is trained to directly predict annotations instead of noise, thereby mitigating harmful variance. Furthermore, they streamline the diffusion process into a single-step procedure to simplify optimization and significantly improve inference speed. Moreover,<remarkably/ notably>, without increasing training data or model capacity significantly,<remarkably/ notably>, Lotus achieves state-of-the-art performance in zero-shot depth and normal estimation across various datasets. Notably,<remarkably/ notably>, Lotus stands out for its efficiency as it outperforms most existing diffusion-based methods in terms of speed. The superior quality and efficiency of Lotus enable a wide range of practical applications such as joint estimation and single/multi-view 3D reconstruction. The findings presented in this paper provide valuable insights into optimizing diffusion-based models for dense prediction tasks and offer promising avenues for future research in this domain. For more information about Lotus and its applications, readers are encouraged to visit the project page at https://lotus3d.github.io/.