Image Generation › Text-to-Image Generation

Generating images from text descriptions requires connecting language to the visual generation process. This is done through conditioning: the noise-prediction network $\boldsymbol{\epsilon}_\theta$ receives the noisy image, the timestep, and a text representation that guides the generation.

The general principle is that the model learns a conditional distribution $p_\theta(\mathbf{x} \mid c)$ where $c$ is the text condition, rather than the unconditional $p_\theta(\mathbf{x})$. Every U-Net forward pass receives both the noisy image and the text embedding, allowing the noise prediction to be guided by the semantic content of the prompt.

The key technique for controlling text-conditioned generation is classifier-free guidance (CFG), introduced by Ho and Salimans. The motivating problem is that while a conditional diffusion model generates images corresponding to the text, the correspondence can be weak: the generated image might include elements from the prompt but not strongly emphasize the distinctive aspects that make the text description specific.

The idea is to train the model in two modes simultaneously: conditioned on text, and unconditioned (with the text randomly dropped out, typically 10-20% of the time by replacing the text conditioning with a null embedding). At inference, you run both modes and interpolate their noise predictions:

$$\tilde{\boldsymbol{\epsilon}}_\theta(\mathbf{x}_t, c) = \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, \varnothing) + w \cdot \left(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, c) - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, \varnothing)\right)$$

where:

• $c$: the text conditioning (encoded prompt)
• $\varnothing$: the null conditioning (empty or dropped text)
• $w$: the guidance scale, controlling how strongly the generation follows the text
• $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, c)$: noise prediction conditioned on text
• $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, \varnothing)$: unconditional noise prediction

When $w = 1$, this reduces to the standard conditioned prediction. When $w > 1$ (typical values are 7-12), the conditional direction is amplified beyond the raw conditional prediction: the model is pushed more strongly toward the text-conditioned manifold. When $w = 0$, the result is fully unconditional.

Why does this work? The term $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, c) - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, \varnothing)$ captures how much the denoising direction changes when conditioned on text versus not. This difference is the direction in noise prediction space that corresponds to "move toward samples that fit the text condition." Scaling this up with large $w$ amplifies the text-driven signal, pushing the generation more aggressively toward images that match the prompt.

The tradeoff is that higher guidance scales reduce diversity. With $w = 12$, the model produces images that very closely match the prompt but all tend to look similar; with $w = 3$, there is more variation at the cost of sometimes weaker prompt adherence. The guidance scale is one of the most important inference-time hyperparameters.

The text conditioning $c$ is typically derived from a pre-trained language model that maps the prompt to a sequence of dense embeddings.

CLIP text encoder: Used in DALL-E 2 and Stable Diffusion 1.x, the CLIP text encoder was trained to align with visual representations through contrastive learning on image-text pairs, as we discussed in the CLIP chapter (Part L). This makes CLIP text embeddings naturally suited for conditioning image generation: they encode language in a space that is semantically linked to visual concepts. The embedding captures semantic meaning in a vision-aware way, which helps the diffusion model bridge language and image.

Large language model encoders: Imagen (Google, 2022) used T5-XXL as its text encoder, a 4.6B parameter language model not specifically trained for visual alignment. Despite lacking CLIP's vision-language grounding, T5 embeddings proved highly effective because T5 handles linguistic structure and negation alongside composition and rare concepts better than CLIP's text encoder. The T5 embeddings carry richer linguistic context, which improves generation of complex, compositional prompts.

Multiple encoder combination: Stable Diffusion XL (SDXL, 2023) and Stable Diffusion 3 (SD3, 2024) use multiple text encoders simultaneously, combining OpenCLIP's ViT-bigG text encoder with the original CLIP ViT-L encoder (in SDXL) or with T5-XXL (in SD3). The outputs are concatenated or pooled together before being fed to the diffusion backbone. This combination captures both vision-aligned CLIP semantics and linguistically rich T5 representations. This provides broader coverage of both visual and textual concepts.

The text is tokenized and passed through the text encoder to produce a sequence of embeddings, one per token. These embeddings are injected into the U-Net through cross-attention layers: the image features at each spatial location act as queries, and the text embeddings serve as keys and values. Each spatial location in the image can thus attend to the text tokens most relevant to what should appear at that location, creating fine-grained text-image alignment. A text token for "red" might be strongly attended to by the image region where the red object should appear.

DALL-E 2 (OpenAI, 2022) took a distinctive two-stage approach to text-to-image generation, motivated by the observation that CLIP image embeddings capture semantic visual content in a rich, structured way.

The pipeline works as follows. First, a prior network generates a CLIP image embedding from the text prompt. This prior is itself a diffusion model (or autoregressive model) that learns the mapping from text embeddings to image embeddings in CLIP's joint space. Given a text prompt describing a scene, the prior generates a CLIP image embedding that represents the visual concept, without yet specifying pixel-level details.

Second, a decoder (a pixel-space diffusion model) generates an image from that CLIP image embedding. Because CLIP image embeddings encode semantic visual content rather than specific pixels, the decoder can focus on rendering visual content consistently with the embedding rather than interpreting language from scratch.

The prior bridges the gap between language and vision space. CLIP's joint embedding space aligns text and images semantically, but a single text prompt corresponds to many possible images. The prior models this one-to-many mapping, sampling different visual interpretations that are all consistent with the text. The result showed strong semantic faithfulness: images clearly corresponded to the described content, even for novel combinations.

Imagen (Google, 2022) demonstrated a surprising lesson: text understanding quality matters as much as image modeling quality for text-to-image generation.

Imagen uses a cascade of diffusion models. A base model generates $64 \times 64$ images conditioned on T5-XXL text embeddings. Two super-resolution diffusion models then upscale the image to $256 \times 256$ and then $1024 \times 1024$, each conditioned on both the original text and the lower-resolution image. The final output is a high-resolution image generated through this pipeline.

The key finding was that doubling the size of the text encoder (using T5-XXL at 4.6B parameters) improved image quality and text adherence more than doubling the size of the image diffusion model. This implies that the bottleneck in text-to-image generation is often the quality of text understanding, not image rendering. A model that does not understand "the woman standing to the left of the red building" will produce a plausible-looking image that fails to respect the spatial and attribute relationships in the text.

Imagen also introduced dynamic thresholding for classifier-free guidance. At high guidance scales, the predicted clean image values can fall outside the valid range $[-1, 1]$, producing oversaturated images. Dynamic thresholding scales the entire prediction by the maximum absolute value if that exceeds a threshold, preserving relative magnitudes while keeping the image in range. This allows using higher guidance scales without oversaturation.

Stable Diffusion (CompVis/Stability AI, 2022) combined latent diffusion with CLIP text conditioning to create the first major open-source text-to-image model. Its architecture became the standard for the open-source ecosystem due to its balance of computational efficiency and image quality.

The key components are:

• A VAE encoder/decoder that compresses $512 \times 512$ images to $64 \times 64 \times 4$ latent codes (spatial compression factor 8, with 4 latent channels)
• A CLIP ViT-L/14 text encoder (frozen) that produces 77 token embeddings of dimension 768
• A latent U-Net that performs diffusion in the compressed $64 \times 64$ latent space, with cross-attention layers throughout for text conditioning
• A linear noise schedule with $T = 1000$ training steps

The open release of model weights enabled a large ecosystem: fine-tuning on specific styles and subjects (DreamBooth, Textual Inversion, LoRA), specialized control adapters (ControlNet, IP-Adapter), GUI tools (AUTOMATIC1111, ComfyUI), and commercial applications. SDXL (2023) scaled this architecture with a larger U-Net and dual CLIP encoders, achieving substantially better image quality and prompt adherence.

Stable Diffusion 3 (2024) moved away from the U-Net architecture toward a Diffusion Transformer (DiT) backbone, using multimodal attention layers that jointly process both image patches and text tokens in a single transformer block, rather than separate processing with cross-attention injection.