Definition

A diffusion model is a type of generative model built on one idea: learn how to gradually restore an image that’s been corrupted with noise, then start from pure random noise and run that restoration over and over until a brand-new image emerges.

Diffusion models stand out for image quality and diversity, and they’ve become the dominant approach for text-to-image, image editing and video generation.

How It Works

A diffusion model involves two processes running in opposite directions:

Forward process (adding noise). Take a real image and add Gaussian noise to it over many small steps, following a fixed schedule. After enough steps, the image becomes almost pure random noise. Nothing is learned here; it’s a fixed mathematical operation.

Reverse process (removing noise). Train a neural network that, at any step, looks at a noisy image and predicts the noise in it (or, equivalently, a cleaner version). During training, you pick a random image, add a random amount of noise, ask the network to predict the noise, and update it based on the error.

To generate a new image, start from pure noise and use the trained network to strip away predicted noise step by step. After a number of steps, you get a clear image.

Several advances made diffusion practical:

Latent diffusion. Running diffusion directly on high-resolution pixels is expensive. Latent diffusion models first compress images into a smaller “latent space” with an autoencoder, run diffusion there, then decode back to pixels, cutting compute dramatically.

Conditioning. Feeding text embeddings into the denoising network, often via cross-attention, steers the output toward the prompt. This is the foundation of text-to-image.

Classifier-free guidance. The model computes predictions both with and without the prompt and amplifies the difference, so the image follows the description more closely. Higher guidance means closer adherence to the prompt, at some cost to diversity and naturalness.

Faster sampling. Early diffusion models needed many steps per image. Later samplers and distillation techniques cut the step count substantially.

Denoising networks were originally mostly U-Nets; more and more recent models use Transformers as the backbone instead.

Typical Applications

  • Text-to-image: illustrations, photo-style images and design sketches from text prompts.
  • Image editing: inpainting, outpainting, style transfer, object removal.
  • Super-resolution and restoration: upscaling low-res images, repairing old photos.
  • Video generation: extending diffusion across time to produce moving sequences.
  • Audio and beyond: speech and music generation, plus research uses like molecular design.
  • Multimodal: text-to-image models must understand both text and images.
  • Transformer: increasingly used as the denoising backbone.
  • Attention mechanism: cross-attention ties image content to the text prompt.
  • Embedding: prompts are converted to vectors before they can guide generation.
  • Generative adversarial networks (GANs): the leading image generation approach before diffusion took over.

Papers: Denoising Diffusion Probabilistic Models, High-Resolution Image Synthesis with Latent Diffusion Models