Image Generation › Summary

This chapter covered the core ideas and practical methods of modern image generation:

• Diffusion models define a forward process that gradually corrupts images with Gaussian noise, then train a neural network to reverse this process step by step, converting pure noise into structured samples
• The closed-form forward process allows computing noisy images at any timestep $t$ directly: $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\, \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\, \boldsymbol{\epsilon}$, making training parallelizable across timesteps
• The simplified training objective trains the network to predict added noise: $\mathcal{L} = \mathbb{E}\left[\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\|^2\right]$, derived from a variational lower bound but simplified for stability
• Latent diffusion moves the process to a compressed VAE latent space, reducing compute by roughly an order of magnitude without sacrificing image quality
• Text-to-image generation uses cross-attention to inject text embeddings into the U-Net at every spatial scale, with classifier-free guidance amplifying the text-conditioned signal at inference to improve prompt adherence
• Sampling efficiency improved from 1000-step DDPM to 20-50-step DDIM and DPM-Solver++ by exploiting the ODE structure of the reverse process
• Image editing techniques including SDEdit, inpainting, InstructPix2Pix, DreamBooth, LoRA, Textual Inversion extend generation to controlled modification of existing images and subject personalization
• ControlNet and IP-Adapter provide spatial and visual conditioning beyond text, enabling precise compositional control
• Evaluation uses FID for distributional quality, CLIP Score for text alignment, and human evaluation for subjective quality; each metric captures different aspects and has known blind spots
• Key limitations include slow sampling, compositional failures, poor text rendering, copyright challenges, memorization, difficult evaluation

Diffusion models represent the current state of the art in image generation. Their core ideas, noise corruption and learned reversal, have already extended beyond static images into video generation (Sora, Runway Gen-3, Stable Video Diffusion), 3D shape synthesis, protein structure generation, audio synthesis, and molecular design. The mathematical framework of learning to denoise at multiple noise scales has proven to be a remarkably flexible and general principle for generative modeling.