Stable Diffusion (2022) › The Solution

Stable Diffusion addressed these challenges through a latent diffusion architecture that dramatically reduced computational requirements while maintaining high image quality. The key insight was that operating in a compressed latent space rather than directly on pixels could reduce computational complexity by orders of magnitude while preserving the information needed for high-quality generation. By compressing images into a latent representation that captured essential visual features in a much smaller space, the diffusion process could work with compressed data that required far less computation to process.

The core innovation of Stable Diffusion was using a variational autoencoder (VAE) to compress images into a latent space where the diffusion process could operate efficiently. The VAE consisted of an encoder that compressed high-resolution images into a latent representation, and a decoder that reconstructed images from latent codes. This compression was not lossless, but the VAE was trained to preserve the visual information necessary for high-quality image generation while dramatically reducing dimensionality.

For a 512x512 pixel image, the VAE might compress it to a 64x64 latent representation, reducing the number of values to process by a factor of 64. Instead of processing over 260,000 pixel values through each diffusion step, the model could work with approximately 4,000 latent values. This compression meant that each step of the diffusion process required far less GPU memory and computation, making it possible to run the model efficiently on consumer GPUs with 8GB or even 6GB of VRAM.

The diffusion process operated in this compressed latent space, producing representations that the VAE could decode into images. A U-Net learned to reverse a noise-adding process, progressively refining random latent noise over multiple steps. Text prompts conditioned each step so the output followed the requested description.

The U-Net's encoder-decoder structure and skip connections captured global structure while preserving local detail. This helped the model align high-level semantic content with local visual features in the generated image.

Text prompts were processed through a text encoder, typically CLIP or a similar model, that converted textual descriptions into embedding vectors. These text embeddings guided the diffusion process at each step, conditioning the noise prediction on the desired output. The model learned to associate text embeddings with visual features in the latent space, enabling it to generate images that matched text descriptions.

Text conditioning allowed Stable Diffusion to process prompts describing the visual style and composition of objects within a scene. Learned relationships between text concepts and visual features supported outputs ranging from photorealistic scenes to conceptual illustrations.

Stable Diffusion was trained on large datasets of image-text pairs, learning the relationships between textual descriptions and visual content. The training process involved multiple components: the VAE learned to compress and reconstruct images, the diffusion model learned to generate latent representations, and the text encoder (or the connections between text and image) learned to associate text with visual features.

The training data included millions of images paired with descriptive text, allowing the model to associate language with visual concepts. Stable Diffusion could vary subject matter and composition across many visual styles while maintaining overall coherence.

Safety measures were incorporated into the training process to reduce the generation of harmful or inappropriate content. These measures included filtering training data, incorporating safety constraints during training, and designing the system to avoid generating certain types of problematic content. While not perfect, these measures represented important steps toward responsible deployment of image generation technology.