Stable Diffusion Paper Explained: Latent Diffusion Models

TL;DR

Latent Diffusion Models (LDMs) move the expensive part of diffusion out of pixel space. Instead of denoising a full-resolution image, an autoencoder first compresses the image into a small latent grid, and the diffusion U-Net learns to generate that — cutting training and sampling cost by an order of magnitude while keeping quality competitive with pixel-space models. A cross-attention layer added to the U-Net turns the same architecture into a general conditional generator for text, layouts, and segmentation maps. This is the architecture Stable Diffusion is built on.

A naming note before we start

The paper is High-Resolution Image Synthesis with Latent Diffusion Models (Rombach et al., CVPR 2022). It never says "Stable Diffusion." That name belongs to the August 2022 open release from Stability AI, CompVis, and Runway — which is an LDM text-to-image model scaled up and trained on a LAION-5B subset. So when someone asks for "the Stable Diffusion paper," this is the right reference, but the honest framing is: Stable Diffusion is a productized, scaled LDM, not a separate method.

The problem

Diffusion models were already winning on image quality by 2021, but they ran directly on pixels. That is brutally expensive. Training a strong pixel-space diffusion model could burn hundreds of GPU-days, and sampling meant running many sequential denoising passes over a high-dimensional pixel tensor.

The waste has a specific cause. A diffusion model trained on pixels spends most of its capacity and compute modeling high-frequency detail that humans barely perceive — the exact texture of individual pixels — rather than the semantic structure that actually determines whether an image looks right. The authors frame this as two distinct regimes: a perceptual compression stage that removes imperceptible detail, and a semantic compression stage where the generative model should be doing its real work. Pixel-space diffusion conflates the two and pays for it.

The goal of the paper is narrow and practical: keep the quality and flexibility of diffusion models, but drastically reduce the compute needed to train and sample them, so high-resolution synthesis isn't locked to industrial GPU clusters.

How it works

LDM splits generation into two stages that are trained separately.

Stage one — perceptual compression. An encoder E maps an image x to a latent z = E(x); a decoder D reconstructs it. This autoencoder is trained with a perceptual loss plus a patch-based adversarial objective, which keeps reconstructions sharp and avoids the blurriness a plain pixel-reconstruction loss would produce. A mild regularizer keeps the latent space well-behaved, and the paper studies two variants:

• KL-reg: a small KL penalty pulling the latent toward a standard normal, similar in spirit to a VAE.
• VQ-reg: a vector-quantization layer inside the decoder.

The latent is spatially downsampled by a factor f. The authors sweep f ∈ {1, 2, 4, 8, 16, 32} and find a clear sweet spot: f = 4 and f = 8 give the best trade-off. Too little compression (f = 1, 2) leaves the diffusion model doing expensive pixel-level work; too much (f = 32) throws away detail the decoder can't recover, capping quality.

Stage two — diffusion in latent space. A time-conditional U-Net learns to denoise in this compact latent space. Because the latent is so much smaller than the image, every training step and every sampling step is far cheaper, while the decoder restores full-resolution perceptual fidelity at the end. The autoencoder is trained once and then reused across many different diffusion models and tasks, which amortizes its cost.

Conditioning via cross-attention. To make the model controllable, the authors add cross-attention layers to the U-Net. A domain-specific encoder turns the conditioning input — a text prompt through a transformer encoder, a semantic map, a layout — into a sequence of tokens, and the U-Net attends to those tokens while denoising. The elegant part is that one mechanism covers text-to-image, layout-to-image, semantic-map-to-image, super-resolution, and inpainting. You swap the conditioning encoder, not the architecture.

Convolutional high-resolution sampling. Because the U-Net is fully convolutional, it generalizes beyond its training resolution. The model can be applied in a sliding/convolutional fashion to produce images larger than anything it saw during training, which is what makes megapixel-scale super-resolution and inpainting practical.

Results

All numbers below are as reported in the paper. Lower FID is better; higher Inception Score (IS) is better.

On class-conditional ImageNet, LDM-4-G beats ADM-G while using substantially fewer parameters and less compute. The CelebA-HQ FID of 5.11 was state of the art at the time. On LSUN-Bedrooms, LDM's 2.95 sits close to ADM's 1.90 — but reaches it with roughly half the parameters and about 4× less training compute, which is the whole point.

The text-to-image result is the headline for what came later: a 1.45B-parameter LDM trained on LAION-400M reaches FID 12.63 on MS-COCO with classifier-free guidance, competitive with much larger autoregressive and diffusion text-to-image models of the era. The paper also reports a new state of the art on inpainting, plus strong semantic-map-to-image and super-resolution (e.g. ImageNet 64→256) results.

On efficiency, the comparison against pixel-based diffusion is the takeaway: roughly a 2.7× (or greater) sampling speed-up alongside an FID improvement of about 1.6×, and training large models becomes feasible on a single GPU rather than a cluster.

Limitations and context

The two-stage design is also the main constraint. Quality is bounded by the pretrained autoencoder: if the downsampling factor is too aggressive, fine detail is lost in z and the diffusion model has no way to recover it. The latent space is a hard fidelity ceiling, not just a convenience.

LDM reduces per-step cost but does not change the fundamental shape of diffusion — sampling is still an iterative, many-step process, and it remains slower than a single forward pass through a GAN. Later work (DDIM, distillation, latent consistency models) attacks that sequential cost directly; this paper does not.

The autoencoder can also introduce subtle reconstruction artifacts, so tasks that demand pixel-exact faithfulness may still prefer pixel-space modeling. And "democratized" is relative. Per-step compute drops sharply, but training the large text-to-image model still required real resources and a web-scale dataset.

That dataset dependency carries the heaviest baggage. Training on large, uncurated, web-scraped collections like LAION raises bias, consent, copyright, and memorization concerns — issues that moved from academic footnote to public controversy once Stable Diffusion shipped on top of this architecture.

Why it matters

LDM is the direct architectural foundation of Stable Diffusion, which made high-quality text-to-image generation runnable on consumer GPUs and became one of the most widely used open generative models. More broadly, the recipe — compress with an autoencoder, then diffuse in the latent space — became the dominant paradigm for generative media and now shows up in latent video diffusion, audio, and 3D.

The cross-attention conditioning design proved equally durable. It set the template for controllable generation, and it is the specific hook that ControlNet, LoRA, DreamBooth, and IP-Adapter all attach to. A large part of the modern open generative-imaging ecosystem is, structurally, this paper plus extensions.

References

• Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022. arXiv:2112.10752
• Official code: github.com/CompVis/latent-diffusion