Image Generation › Diffusion Models: The Core Idea

The fundamental insight of diffusion models is this: if you have a process that gradually destroys structure, you can learn its reverse to build structure back.

Consider what happens when you gradually add Gaussian noise to an image. At first, the image is recognizable but slightly grainy. After more noise is added, the details blur and only global structure remains. Continue adding noise, and eventually you have something indistinguishable from pure random static, regardless of what the original image was. This forward process destroys information in a principled, mathematically tractable way.

Now flip the perspective. If you could learn to undo each small noise step, you could start with pure random noise and gradually recover a coherent image. You do not need to generate a complete image in one shot; you only need to denoise slightly at each step. This is dramatically more tractable because each denoising step is a small, local change to a partially formed image, rather than a jump from noise to image.

The forward process (also called the diffusion process) takes a real image $\mathbf{x}_0$ and gradually adds Gaussian noise over $T$ timesteps:

$$q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}\left(\mathbf{x}_t;\, \sqrt{1 - \beta_t}\, \mathbf{x}_{t-1},\, \beta_t \mathbf{I}\right)$$

where:

• $\mathbf{x}_t$: the noisy image at timestep $t$
• $\mathbf{x}_{t-1}$: the image at the previous (less noisy) timestep
• $\beta_t$: the noise schedule at timestep $t$, controlling how much noise is added
• $\mathcal{N}(\mu, \sigma^2)$: a Gaussian distribution with mean $\mu$ and variance $\sigma^2 \mathbf{I}$

Each step scales down the signal by $\sqrt{1 - \beta_t}$ (to preserve approximate unit variance) and adds Gaussian noise with variance $\beta_t$. The scaling factor ensures that the overall variance of $\mathbf{x}_t$ remains roughly constant over time: as signal energy decreases, noise energy increases by the same amount. With a carefully designed noise schedule, after $T$ steps (typically $T = 1000$), $\mathbf{x}_T$ is approximately pure Gaussian noise, regardless of what $\mathbf{x}_0$ was.

A mathematical property makes diffusion computationally tractable: you can compute the noisy image at any arbitrary timestep $t$ directly from $\mathbf{x}_0$, without simulating each intermediate step. This is possible because Gaussian distributions have the property that the composition of Gaussians is itself Gaussian. Define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$ (the cumulative product of all $\alpha$ values up to step $t$). Then:

$$q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}\left(\mathbf{x}_t;\, \sqrt{\bar{\alpha}_t}\, \mathbf{x}_0,\, (1 - \bar{\alpha}_t) \mathbf{I}\right)$$

which means we can sample $\mathbf{x}_t$ directly as:

$$\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\, \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\, \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$

where $\boldsymbol{\epsilon}$ is pure Gaussian noise. This "reparameterization" has major practical implications. During training, you do not need to simulate the full forward chain from $t = 0$ to $t = T$ for every training example. Instead, for each training image you pick a random timestep $t$, sample noise $\boldsymbol{\epsilon}$, compute the corresponding $\mathbf{x}_t$ using the formula above, and train the model on that single noisy image. This makes training embarrassingly parallel across timesteps.

The noise schedule $\{\beta_t\}_{t=1}^{T}$ controls how quickly structure is destroyed as noise is added. Its design has a significant impact on the quality of the learned model.

A natural first choice is a linear schedule: $\beta_t$ increases linearly from a small value (like $10^{-4}$) to a larger value (like $0.02$). This is easy to understand, but it has a practical flaw: it destroys too much structure too quickly at low noise levels, where most of the perceptual information lives, and too little at high noise levels. The model ends up spending many timesteps learning to denoise nearly-Gaussian noise, which carries little useful gradient signal for learning image structure.

The cosine schedule, proposed by Nichol and Dhariwal, addresses this by defining $\bar{\alpha}_t$ directly rather than working through $\beta_t$:

$$\bar{\alpha}_t = \frac{f(t)}{f(0)}, \quad f(t) = \cos\left(\frac{t/T + s}{1 + s} \cdot \frac{\pi}{2}\right)^2$$

where:

• $T$: the total number of timesteps
• $s$: a small offset (typically 0.008) that prevents $\beta_t$ from being too large near $t = 0$
• The cosine function ensures $\bar{\alpha}_t$ decreases smoothly from 1 to near 0 following a cosine curve

The cosine schedule keeps $\bar{\alpha}_t$ near 1 for a longer initial period before dropping more smoothly to near zero at the end. This allocates more timesteps to informative intermediate noise levels, where the model can learn meaningful structure. Empirically, cosine schedules produce better image quality than linear ones, especially for higher-resolution images.

More recent work has explored learned noise schedules (optimizing $\{\beta_t\}$ as parameters) and continuous-time diffusion (defining the noise process as a stochastic differential equation rather than discrete steps), but linear and cosine schedules remain widely used because of their simplicity and reliable performance.

The reverse process aims to invert the forward process: starting from pure noise $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$, progressively denoise to recover a clean image $\mathbf{x}_0$. The true reverse conditional distribution $q(\mathbf{x}_{t-1} \mid \mathbf{x}_t)$ exists but is intractable to compute directly, because it requires integrating over all possible clean images $\mathbf{x}_0$ consistent with the noisy image $\mathbf{x}_t$. However, if we condition on the clean image, the reverse conditional becomes tractable:

$$q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0) = \mathcal{N}\left(\mathbf{x}_{t-1};\, \tilde{\boldsymbol{\mu}}_t(\mathbf{x}_t, \mathbf{x}_0),\, \tilde{\beta}_t \mathbf{I}\right)$$

where $\tilde{\beta}_t = \frac{(1 - \bar{\alpha}_{t-1})}{1 - \bar{\alpha}_t} \beta_t$ is the posterior variance. This formula gives the optimal denoising direction when the clean image $\mathbf{x}_0$ is known. In practice, we do not know $\mathbf{x}_0$ during generation, so we learn a neural network $p_\theta$ to approximate the reverse conditional:

$$p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t) = \mathcal{N}\left(\mathbf{x}_{t-1};\, \boldsymbol{\mu}_\theta(\mathbf{x}_t, t),\, \sigma_t^2 \mathbf{I}\right)$$

where $\boldsymbol{\mu}_\theta$ is a neural network parameterized by $\theta$ that predicts the mean of the reverse distribution.

Rather than directly predicting the mean, most implementations train the network to predict the noise $\boldsymbol{\epsilon}$ that was added. This is the $\boldsymbol{\epsilon}$-prediction formulation. Given the predicted noise $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$, the reverse step mean becomes:

$$\boldsymbol{\mu}_\theta(\mathbf{x}_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right)$$

where:

• $\frac{1}{\sqrt{\alpha_t}}$: un-scales the signal by reversing the scaling applied in the forward step
• $\frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$: subtracts the predicted noise contribution, pushing the estimate toward the clean image

The $\boldsymbol{\epsilon}$-prediction formulation is used in practice because it produces stable training dynamics: the loss is well-scaled across all timesteps, and the noise prediction task is a well-defined, bounded regression problem. An alternative is $\mathbf{x}_0$-prediction (predicting the clean image directly), which is sometimes more stable at low noise levels, and $v$-prediction (predicting a linear combination of noise and signal), which was introduced for improved training at high resolutions. Modern models often blend these formulations.

The full diffusion training objective is derived from a variational lower bound on the log-likelihood of the data. Maximizing $\log p_\theta(\mathbf{x}_0)$ directly is intractable, but we can instead maximize a lower bound that decomposes into per-timestep KL divergences between the learned reverse process and the true posterior. Ho et al. (2020) showed that a simplified version of this objective works just as well in practice:

$$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}}\left[\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta\!\left(\sqrt{\bar{\alpha}_t}\, \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\, \boldsymbol{\epsilon},\, t\right)\right\|^2\right]$$

where:

• $t \sim \text{Uniform}(1, T)$: a uniformly randomly sampled timestep
• $\mathbf{x}_0$: a real training image sampled from the training dataset
• $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$: noise sampled from a standard Gaussian
• $\sqrt{\bar{\alpha}_t}\, \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\, \boldsymbol{\epsilon}$: the noisy image at timestep $t$ using the reparameterization trick
• $\boldsymbol{\epsilon}_\theta$: the neural network predicting the noise from the noisy image and timestep

In words: pick a random training image, pick a random timestep, add the corresponding amount of noise, and train the network to predict what noise was added. This is a denoising task framed differently at each timestep. At high $t$, the image is mostly noise and the model must infer global structure. At low $t$, the image is mostly clean and the model must predict the residual fine-grained noise.

The simplification from the full variational lower bound to $\mathcal{L}_{\text{simple}}$ drops per-timestep weighting that appears in the variational objective, trading theoretical optimality for empirical training stability. In practice, models trained with $\mathcal{L}_{\text{simple}}$ generate better images than those trained with the full variational bound, likely because the uniform weighting prevents some timesteps from dominating the gradient signal.

The neural network $\boldsymbol{\epsilon}_\theta$ needs to accept a noisy image and a timestep, and output a same-sized noise prediction. The dominant architecture is a U-Net, originally developed by Ronneberger et al. for biomedical image segmentation, adapted for the diffusion setting.

The U-Net's key structural property is its encoder-decoder architecture with skip connections. The encoder path progressively downsamples the image using strided convolutions or pooling, building a hierarchy of features from fine-grained local patterns to coarse global structure. The decoder path progressively upsamples back to the original resolution using transposed convolutions or bilinear upsampling. Skip connections copy feature maps directly from each encoder stage to the corresponding decoder stage, allowing fine-grained spatial details to flow directly to the decoder. Without skip connections, the encoder bottleneck would force all spatial information through the narrow representation, causing loss of fine detail.

For diffusion, the U-Net is adapted with several modifications:

• Timestep embeddings: the scalar timestep $t$ is embedded via sinusoidal positional encodings (similar to those we discussed in Part XIV on positional encoding) and projected through small feed-forward networks. These embeddings are added to the feature maps at each resolution level, allowing the network to know at which noise level it is operating.
• Residual blocks: each U-Net block wraps its core convolutions in residual connections, improving gradient flow during training and allowing deeper networks.
• Self-attention layers: attention layers are added at lower resolutions where the spatial dimensions are small, making attention computationally feasible. These allow the network to model long-range dependencies between distant spatial locations. A global composition constraint, for example a sky being blue and grass being green, requires correlating distant pixels.
• Group normalization: normalization is applied within feature groups rather than across batch dimensions, which is more stable for the small batch sizes used in diffusion training.

The result is a flexible model that can process images at their full resolution while conditioning on the noise level. Because the same network handles all $T$ timesteps, it learns a range of denoising behaviors: at high noise (early in generation), it focuses on global structure and low-frequency signals; at low noise (late in generation), it refines fine details and textures. The model implicitly learns a curriculum from coarse to fine.