Image Generation › Image Editing

Beyond generating new images from scratch, diffusion models can modify existing images in semantically meaningful ways. This is possible because the forward and reverse processes can be applied to noise and to any image.

SDEdit (Meng et al., 2021) is the simplest editing approach. You take a reference image, add noise to a specific level $t^*$, then run the reverse process with conditioning. This approach does not require any training changes; it uses a pre-trained diffusion model directly.

The noise level $t^*$ controls the tradeoff between fidelity to the original and freedom to change. High $t^*$ means more noise is added, so the model can make larger structural changes but may deviate substantially from the original content. Low $t^*$ preserves the original closely but allows only subtle texture or color edits. This makes $t^*$ an intuitive dial: setting it at $t^* = 0.5T$ keeps the rough composition and changes details, while $t^* = 0.8T$ allows more substantial changes to shape and content.

SDEdit works well for sketch-based generation (start with a rough hand-drawn sketch, add noise, condition on text), style transfer (starting from a content image), and small targeted edits. Its simplicity is both its strength (no additional training needed) and its limitation (it cannot make precise, localized edits).

Inpainting fills a masked region of an image while maintaining consistency with the surrounding context. This is useful for removing unwanted objects, filling in damaged areas, or adding new content to existing images.

During the reverse process, at each timestep, the unmasked regions of the noisy image are replaced with the known noisy image at that level:

$$\mathbf{x}_t^{\text{inpaint}} = m \odot \mathbf{x}_t^{\text{known}} + (1 - m) \odot \mathbf{x}_t^{\text{generated}}$$

where:

• $m$: the binary mask (1 for pixels to keep, 0 for pixels to generate)
• $\mathbf{x}_t^{\text{known}}$: the original image noised to level $t$ using the forward process
• $\mathbf{x}_t^{\text{generated}}$: the image generated so far by the reverse process
• $\odot$: element-wise multiplication

At each reverse step, the generated regions are denoised by the network while the known regions are reset to their noisy versions (forward-noised at exactly timestep $t$). This ensures that in the final output, the known regions are exactly the original pixels and the generated region is produced by the diffusion model under the constraint of fitting smoothly with the surroundings.

A limitation of this naive approach is that the seam between generated and preserved regions can be sharp, because the generated region is optimized without full awareness of the boundary constraint. Improved inpainting models are fine-tuned with masked training examples, teaching the model to explicitly condition on both the unmasked image content and the generation task at the boundary.

Rather than conditioning on a masked region, InstructPix2Pix (Brooks et al., 2022) conditions on both an original image and a text instruction describing the desired edit, such as "make the sky sunset" or "add glasses to the person."

Training this model required a creative data generation pipeline, since pairs of (original image, edit instruction, edited image) are not naturally available at scale. The solution was to synthesize training data: GPT-3 generated pairs of (image caption, edit instruction, edited image caption), and a text-to-image model (DALL-E) generated image pairs from the caption pairs. The diffusion model was then trained to perform the transformation given both the original image and the instruction.

At inference, InstructPix2Pix uses a modified CFG that combines guidance from both the text instruction and the original image, allowing control over how much the edit changes the image versus how literally the instruction is followed.

A related class of techniques personalizes image generation to specific subjects. DreamBooth (Ruiz et al., 2022) fine-tunes the entire diffusion model on 3-20 images of a specific person, object, or style. The appearance is bound to a rare token, like "sks dog," that is unlikely to appear in the original training data. After fine-tuning, prompts containing that token generate the specific subject in novel contexts: "a photo of sks dog in the snow" produces the specific dog in a snowy scene.

The challenge with full fine-tuning is the risk of language drift: the model may overfit to the specific images and forget other concepts. DreamBooth addresses this with a prior preservation loss that continues training on text-generated images of the general class (regular dogs, in this example) alongside the specific subject images.

Low-Rank Adaptation (LoRA), which we covered in Part XXXV on parameter-efficient fine-tuning, addresses both the cost and drift concerns by fine-tuning only small low-rank matrices inserted into the attention layers. Instead of updating all weight matrices, LoRA decomposes the update as $\Delta W = AB$ where $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times d}$ with small rank $r$ (typically 4-16). LoRA files for Stable Diffusion are typically 50-200 MB rather than the several GB of full model weights, and they can be combined by summing their contributions with different weights, allowing compositional style mixing.

Textual Inversion (Gal et al., 2022) takes an even lighter-weight approach: rather than changing the model weights at all, it learns a new token embedding that represents the target concept. The denoising network is entirely frozen; only a new token embedding $v^*$ is optimized to minimize the denoising loss on the reference images:

$$\mathcal{L} = \mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}}\left[\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta\!\left(\mathbf{x}_t,\, c(S^*)\right)\right\|^2\right]$$

where $c(S^*)$ is the text conditioning derived from a string $S^*$ containing the new learned token $v^*$. The learned token lives in the same embedding space as the model's existing vocabulary, so it can be combined with other words naturally.

Textual Inversion requires even fewer parameters than LoRA (a single token embedding vector rather than low-rank weight matrices) but captures concepts less flexibly because it is limited to the expressiveness of the fixed embedding space. Complex concepts that require weight adaptations to represent well are better suited to LoRA or DreamBooth.