Image Generation › From GANs to Diffusion: A Brief History

Before diffusion models, the dominant paradigm for image generation was the Generative Adversarial Network (GAN), introduced by Goodfellow et al. in 2014. GANs set up a game between two neural networks: a generator that produces fake images and a discriminator that tries to distinguish real from fake. The generator learns by fooling the discriminator; the discriminator improves by catching the generator. In equilibrium, the generator produces realistic images.

GANs were powerful but notoriously difficult to train. They suffered from mode collapse, a failure mode where the generator produces only a narrow variety of images despite being trained on diverse data. A generator that outputs a single convincing image of a golden retriever technically fools the discriminator, but it has not learned the full distribution. Training instability was equally problematic: the generator and discriminator could fall into unproductive cycles, with the generator oscillating between certain artifact patterns rather than converging. Researchers spent years developing techniques like progressive growing (training first at low resolution and gradually increasing), spectral normalization, gradient penalties, and careful architectural choices just to stabilize training.

Variational Autoencoders (VAEs) offered a different approach. A VAE encodes images into a compact latent distribution, parameterized by a mean and variance, then samples from that distribution to decode new images. VAEs were stable to train and provided an explicit latent space suitable for interpolation and manipulation. However, they produced notoriously blurry outputs. The reason is their reconstruction loss: optimizing pixel-level mean squared error encourages the decoder to average over plausible reconstructions, producing a blurry mean rather than any specific sharp image. The latent bottleneck adds to this averaging pressure by compressing multiple plausible outputs into a single latent point.

Autoregressive models like PixelCNN and VQ-VAE-based approaches (DALL-E v1, ImageGPT) took a third path: model the joint pixel distribution as a product of conditionals, generating one pixel (or patch, or token) at a time. These models can generate very high-quality images but are extremely slow at inference time because generation is sequential. VQ-VAE with a transformer prior (used in the original DALL-E) combined discrete tokenization of images with autoregressive modeling, allowing text-conditioned generation, but still required hundreds to thousands of forward passes.

Diffusion models emerged as an alternative that combined the stability of VAE training with quality approaching or exceeding GANs. Instead of a generator-discriminator game or an encoder-decoder compression, diffusion models learn to reverse a gradual noise process. This framing turns image generation into a sequence of denoising steps, each more tractable than generating an image from scratch. The main point is elegantly simple: if you can reliably remove a small amount of noise, you can chain those removals together to generate images from nothing.