Image Generation › Sampling and Inference

Training a diffusion model gives us the denoising network, but generating images requires running the reverse process. This is the sampling procedure, and the design of samplers has been an active area of research because the original DDPM sampler was far too slow for practical use.

The original Denoising Diffusion Probabilistic Models (DDPM) sampler runs the full $T$ reverse steps:

$$\mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right) + \sigma_t \mathbf{z}$$

where:

• $\frac{1}{\sqrt{\alpha_t}}$: the inverse scaling factor for this step
• $\frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_\theta$: the denoising correction based on the predicted noise
• $\sigma_t \mathbf{z}$: additional stochastic noise injected at each step, where $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$

With $T = 1000$, generating one image requires 1000 neural network forward passes through the full U-Net. At 8-30 milliseconds per forward pass on a modern GPU, this translates to 8-30 seconds per image, far too slow for interactive use.

Denoising Diffusion Implicit Models (DDIM), introduced by Song et al. (2021), made sampling dramatically faster by reformulating the reverse process as a deterministic ordinary differential equation (ODE) rather than a stochastic process.

The key insight is that many different reverse processes can share the same marginals $q(\mathbf{x}_t \mid \mathbf{x}_0)$ as DDPM while taking larger, more efficient steps. DDIM defines a non-Markovian reverse process that allows skipping many timesteps. Instead of going through all 1000 timesteps in order, DDIM can take evenly spaced steps visiting only 20 timesteps out of 1000, while still producing high-quality images.

The deterministic nature of DDIM has an additional useful property: it acts as an encoder as well as a decoder. You can run the DDIM forward process on a real image to find the latent noise representation corresponding to that image (called DDIM inversion), then modify the generation process at that latent to edit the image. This is not possible with stochastic DDPM because the stochasticity makes inversion non-invertible.

Subsequent work formalized diffusion sampling as solving a differential equation and applied numerical ODE analysis to improve it. DPM-Solver and its successor DPM-Solver++ (Lu et al., 2022) treat the reverse process as an ODE with a known analytical structure and apply higher-order solvers. These can produce high-quality images in as few as 10-20 steps by taking larger, more accurate integration steps.

The key insight is that the diffusion reverse process has a semi-linear structure: the linear component (corresponding to Gaussian noise removal) can be solved analytically, and only the nonlinear component (the U-Net's neural denoising) needs to be approximated numerically. By exploiting this structure, DPM-Solver takes each integration step far more accurately than DDIM, which uses a simpler first-order approximation.

Other popular samplers include PNDM (Pseudo Numerical Methods for Diffusion Models), Euler and Euler a (ancestral Euler methods with optional stochasticity), and Heun (a second-order Runge-Kutta variant). The choice of sampler affects speed and the character of the output: deterministic samplers give reproducible results from a fixed seed, while stochastic samplers add variation and can sometimes escape local optima in the generation trajectory.