Image Generation › Generation Control and Conditioning

Beyond text, diffusion models can be conditioned on many other signals to give precise control over the output. This is important for professional workflows where the artist needs more control than text alone provides.

ControlNet (Zhang and Agrawala, 2023) adds spatial conditioning signals such as edge maps, pose estimates (OpenPose skeleton), depth maps, surface normal maps, segmentation masks, or text line positions. The architecture addresses a key challenge: how to add conditioning without retraining the entire model.

ControlNet creates a trainable copy of the U-Net encoder. This copy processes the conditioning input (an edge map, for example) and adds its activations to the corresponding layers of the original U-Net's skip connections. The original U-Net is frozen; only the encoder copy and its connection weights are trained.

This design preserves the original U-Net's learned image priors while adding spatial control. Because the conditioning signal is spatial (the same height and width as the image), the network can precisely control image composition.

OpenPose conditioning generates humans in specific poses by feeding a skeleton image as the control signal. Canny edge map conditioning generates images that respect precise boundaries. Depth conditioning preserves 3D structure from a reference image while changing textures or style. This is particularly valuable for architectural rendering (conditioning on floor plans or line drawings), character animation (conditioning on pose sequences from video), and product photography (conditioning on the shape of a product).

Multiple ControlNet models can be stacked, applying combined conditioning. A generation might use edge conditioning for structural fidelity and pose conditioning for character positioning simultaneously.

IP-Adapter (Ye et al., 2023) conditions the diffusion model on a reference image's visual style or content, rather than or in addition to a text prompt. You provide an image whose style, subject, or composition you want to replicate in new generations.

IP-Adapter encodes the reference image with a vision encoder (typically CLIP's visual encoder) and injects those embeddings through a separate cross-attention pathway running in parallel to the existing text cross-attention. The image features serve as additional keys and values alongside the text features, with learnable weights determining the balance between text and image conditioning.

This allows tasks that are difficult to specify through text alone: style transfer (generate this scene in the visual style of this reference painting), face consistency (generate different scenarios featuring a specific person's face), and product placement (generate marketing images featuring a specific product). Because IP-Adapter is an adapter added on top of the frozen base model, it is compatible with any ControlNet adapters and LoRA fine-tuned models for the same base.