SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generation

   Clement Chadebec, Onur Tasar, Eyal Benaroche et al. · 2024 · Computer Vision and Pattern Recognition · cited 75 times · Not summarized yet

   Abstract

   In this paper, we propose an efficient, fast, and versatile distillation method to accelerate the generation of pre-trained diffusion models: Flash Diffusion. The method reaches state-of-the-art performances in terms of FID and CLIP-Score for few steps image generation on the COCO2014 and COCO2017 datasets, while requiring only several GPU hours of training and fewer trainable parameters than existing methods. In addition to its efficiency, the versatility of the method is also exposed across several tasks such as text-to-image, inpainting, face-swapping, super-resolution and using different backbones such as UNet-based denoisers (SD1.5, SDXL) or DiT (Pixart-$\alpha$), as well as adapters. I…

2. Very close Cited by this paper A method it uses Not relevant

   SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

   Dustin Podell, Zion English, Kyle Lacey et al. · 2023 · Computer Vision and Pattern Recognition · cited 5,712 times · AI summary available

   Abstract

   We present SDXL, a latent diffusion model for text-to-image synthesis. Compared to previous versions of Stable Diffusion, SDXL leverages a three times larger UNet backbone: The increase of model parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder. We design multiple novel conditioning schemes and train SDXL on multiple aspect ratios. We also introduce a refinement model which is used to improve the visual fidelity of samples generated by SDXL using a post-hoc image-to-image technique. We demonstrate that SDXL shows drastically improved performance compared the previous versions of Stable Diffusion and achieves results comp…

3. Very close Cited by this paper Not relevant

   High-Resolution Image Synthesis with Latent Diffusion Models

   Robin Rombach, Andreas Blattmann, Dominik Lorenz et al. · 2021 · Computer Vision and Pattern Recognition · cited 28,796 times · AI summary available

   Abstract

   By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to p…

4. Very close Not relevant

   SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers

   Enze Xie, Junsong Chen, Junyu Chen et al. · 2024 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 381 times · AI summary available

   Abstract

   We introduce \model, a text-to-image framework that can efficiently generate images up to 4096$\times$4096 resolution. \model can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU. Core designs include: (1) Deep compression autoencoder: unlike traditional AEs, which compress images only 8$\times$, we trained an AE that can compress images 32$\times$, effectively reducing the number of latent tokens. (2) Linear DiT: we replace all vanilla attention in DiT with linear attention, which is more efficient at high resolutions without sacrificing quality. (3) Decoder-only text encoder: we replaced T5 with modern dec…

5. Very close Not relevant

   Q-Sched: Pushing the Boundaries of Few-Step Diffusion Models with Quantization-Aware Scheduling

   Natalia Frumkin, Diana Marculescu · 2025 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Text-to-image diffusion models are computationally intensive, often requiring dozens of forward passes through large transformer backbones. For instance, Stable Diffusion XL generates high-quality images with 50 evaluations of a 2.6B-parameter model, an expensive process even for a single batch. Few-step diffusion models reduce this cost to 2-8 denoising steps but still depend on large, uncompressed U-Net or diffusion transformer backbones, which are often too costly for full-precision inference without datacenter GPUs. These requirements also limit existing post-training quantization methods that rely on full-precision calibration. We introduce Q-Sched, a new paradigm for post-training quan…

6. Very close Not relevant

   Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LIV

   Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells et al. · 2023 · European Conference on Computer Vision · cited 10 times · Not summarized yet

   Abstract

   Text-to-image (T2I) generation with Stable Diffusion models (SDMs) involves high computing demands due to billion-scale parameters. To enhance efficiency, recent studies have reduced sampling steps and applied network quantization while retaining the original architectures. The lack of architectural reduction attempts may stem from worries over expensive retraining for such massive models. In this work, we uncover the surprising potential of block pruning and feature distillation for low-cost general-purpose T2I. By removing several residual and attention blocks from the U-Net of SDMs, we achieve 30%~50% reduction in model size, MACs, and latency. We show that distillation retraining is effe…

7. Very close Not relevant

   Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think

   Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt et al. · 2024 · Computer Vision and Pattern Recognition · cited 109 times · AI summary available

   Abstract

   Recent work showed that large diffusion models can be reused as highly precise monocular depth estimators by casting depth estimation as an image-conditional image generation task. While the proposed model achieved state-of-the-art results, high computational demands due to multi-step inference limited its use in many scenarios. In this paper, we show that the perceived inefficiency was caused by a flaw in the inference pipeline that has so far gone unnoticed. The fixed model performs comparably to the best previously reported configuration while being more than 200$\times$ faster. To optimize for downstream task performance, we perform end-to-end fine-tuning on top of the single-step model …

8. Very close Cited by this paper A method it uses Not relevant

   Progressive Knowledge Distillation Of Stable Diffusion XL Using Layer Level Loss

   Yatharth Gupta, Vishnu V. Jaddipal, Harish Prabhala et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 71 times · Not summarized yet

   Abstract

   Stable Diffusion XL (SDXL) has become the best open source text-to-image model (T2I) for its versatility and top-notch image quality. Efficiently addressing the computational demands of SDXL models is crucial for wider reach and applicability. In this work, we introduce two scaled-down variants, Segmind Stable Diffusion (SSD-1B) and Segmind-Vega, with 1.3B and 0.74B parameter UNets, respectively, achieved through progressive removal using layer-level losses focusing on reducing the model size while preserving generative quality. We release these models weights at https://hf.co/Segmind. Our methodology involves the elimination of residual networks and transformer blocks from the U-Net structu…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Adversarial Diffusion Distillation

   Axel Sauer, Dominik Lorenz, Andreas Blattmann et al. · 2023 · Computer Vision and Pattern Recognition · cited 960 times · AI summary available

   Abstract

   We introduce Adversarial Diffusion Distillation (ADD), a novel training approach that efficiently samples large-scale foundational image diffusion models in just 1-4 steps while maintaining high image quality. We use score distillation to leverage large-scale off-the-shelf image diffusion models as a teacher signal in combination with an adversarial loss to ensure high image fidelity even in the low-step regime of one or two sampling steps. Our analyses show that our model clearly outperforms existing few-step methods (GANs, Latent Consistency Models) in a single step and reaches the performance of state-of-the-art diffusion models (SDXL) in only four steps. ADD is the first method to unlock…

2. Close A method it uses Not relevant

   SDXL-Lightning: Progressive Adversarial Diffusion Distillation

   Shanchuan Lin, An-Ran Wang, Xiao Yang · 2024 · arXiv.org · preprint, not peer reviewed · cited 311 times · Not summarized yet

   Abstract

   We propose a diffusion distillation method that achieves new state-of-the-art in one-step/few-step 1024px text-to-image generation based on SDXL. Our method combines progressive and adversarial distillation to achieve a balance between quality and mode coverage. In this paper, we discuss the theoretical analysis, discriminator design, model formulation, and training techniques. We open-source our distilled SDXL-Lightning models both as LoRA and full UNet weights.

3. Close A method it uses Not relevant

   Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference

   Simian Luo, Yiqin Tan, Long-Bo Huang et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 1,057 times · Not summarized yet

   Abstract

   Latent Diffusion models (LDMs) have achieved remarkable results in synthesizing high-resolution images. However, the iterative sampling process is computationally intensive and leads to slow generation. Inspired by Consistency Models (song et al.), we propose Latent Consistency Models (LCMs), enabling swift inference with minimal steps on any pre-trained LDMs, including Stable Diffusion (rombach et al). Viewing the guided reverse diffusion process as solving an augmented probability flow ODE (PF-ODE), LCMs are designed to directly predict the solution of such ODE in latent space, mitigating the need for numerous iterations and allowing rapid, high-fidelity sampling. Efficiently distilled fro…

4. Close A method it uses Not relevant

   InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation

   Xingchao Liu, Xiwen Zhang, Jianzhu Ma et al. · 2023 · Machine Learning · cited 468 times · AI summary available

   Abstract

   Diffusion models have revolutionized text-to-image generation with its exceptional quality and creativity. However, its multi-step sampling process is known to be slow, often requiring tens of inference steps to obtain satisfactory results. Previous attempts to improve its sampling speed and reduce computational costs through distillation have been unsuccessful in achieving a functional one-step model. In this paper, we explore a recent method called Rectified Flow, which, thus far, has only been applied to small datasets. The core of Rectified Flow lies in its \emph{reflow} procedure, which straightens the trajectories of probability flows, refines the coupling between noises and images, an…

5. Close A method it uses Not relevant

   Denoising Diffusion Probabilistic Models

   Jonathan Ho, Ajay Jain, Pieter Abbeel · 2020 · Machine Learning · cited 37,378 times · AI summary available

   Abstract

   We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar…

Papers that cite it

1. Close Uses this paper in depth Not relevant

   DiffServe: Efficiently Serving Text-to-Image Diffusion Models with Query-Aware Model Scaling

   Sohaib Ahmad, Qizheng Yang, Hao-Liang Wang et al. · 2024 · Conference on Machine Learning and Systems · cited 20 times · Not summarized yet

   Abstract

   Text-to-image generation using diffusion models has gained increasing popularity due to their ability to produce high-quality, realistic images based on text prompts. However, efficiently serving these models is challenging due to their computation-intensive nature and the variation in query demands. In this paper, we aim to address both problems simultaneously through query-aware model scaling. The core idea is to construct model cascades so that easy queries can be processed by more lightweight diffusion models without compromising image generation quality. Based on this concept, we develop an end-to-end text-to-image diffusion model serving system, DiffServe, which automatically construct…

2. Close Not relevant

   Adversarial Diffusion Compression for Real-World Image Super-Resolution

   Bin Chen, Gehui Li, Rong-Yuan Wu et al. · 2024 · Computer Vision and Pattern Recognition · cited 75 times · Not summarized yet

   Abstract

   Real-world image super-resolution (Real-ISR) aims to reconstruct high-resolution images from low-resolution inputs degraded by complex, unknown processes. While many Stable Diffusion (SD)-based Real-ISR methods have achieved remarkable success, their slow, multi-step inference hinders practical deployment. Recent SD-based one-step networks like OSEDiff and S3Diff alleviate this issue but still incur high computational costs due to their reliance on large pretrained SD models. This paper proposes a novel Real-ISR method, AdcSR, by distilling the one-step diffusion network OSEDiff into a streamlined diffusion-GAN model under our Adversarial Diffusion Compression (ADC) framework. We meticulousl…

3. Close Uses this paper in depth Not relevant

   David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training

   Wei-Jian Luo, Colin Zhang, De-Bing Zhang et al. · 2024 · International Conference on Machine Learning · cited 10 times · Not summarized yet

   Abstract

   We propose Diff-Instruct* (DI*), a data-efficient post-training approach for one-step text-to-image generative models to improve its human preferences without requiring image data. Our method frames alignment as online reinforcement learning from human feedback (RLHF), which optimizes the one-step model to maximize human reward functions while being regularized to be kept close to a reference diffusion process. Unlike traditional RLHF approaches, which rely on the Kullback-Leibler divergence as the regularization, we introduce a novel general score-based divergence regularization that substantially improves performance as well as post-training stability. Although the general score-based RLHF…

4. Close Not relevant

   AddSR: Accelerating Diffusion-based Blind Super-Resolution with Adversarial Diffusion Distillation

   Ying Tai, Rui Xie, Chen Zhao et al. · 2024 · Pattern Recognition · cited 80 times · Not summarized yet

   Abstract

   Blind super-resolution methods based on stable diffusion showcase formidable generative capabilities in reconstructing clear high-resolution images with intricate details from low-resolution inputs. However, their practical applicability is often hampered by poor efficiency, stemming from the requirement of thousands or hundreds of sampling steps. Inspired by the efficient adversarial diffusion distillation (ADD), we design~\name~to address this issue by incorporating the ideas of both distillation and ControlNet. Specifically, we first propose a prediction-based self-refinement strategy to provide high-frequency information in the student model output with marginal additional time cost. Fur…

5. Close Not relevant

   Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra

   Yoichi Ochiai · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   While real-time image generation using diffusion models has advanced rapidly on NVIDIA GPUs, systematic optimization research on non-CUDA platforms such as Apple Silicon remains extremely limited. In this study, we conducted comprehensive optimization experiments across 10 phases targeting the Apple M3 Ultra (60-core GPU, 512 GB unified memory) with the goal of achieving real-time camera img2img transformation. We explored a wide range of techniques including CoreML conversion, quantization, Token Merging, Neural Engine utilization, compact model exploration, frame interpolation, kNN search-based synthesis, pix2pix-turbo, optical flow frame skipping, and knowledge distillation, quantitativel…