DALL·E 2 (2022) › The Problem

The original DALL·E, released in January 2021, had demonstrated that transformers could generate images from text prompts, but the approach faced several fundamental challenges that limited its practical utility. The autoregressive generation process, which generated images pixel by pixel following a raster scan pattern, struggled to maintain global coherence across the entire image. This meant that while individual regions might look plausible, the overall composition could lack consistency or exhibit artifacts that made the images clearly artificial.

The original DALL·E also lacked the ability to edit images after generation. Users who wanted to adjust an image or explore variations had to regenerate it from scratch with a modified prompt. This iterative refinement process was time-consuming and often failed to produce the desired result, as small changes to text prompts could lead to dramatically different outputs that lost desirable aspects of the original generation.

The image quality of the original DALL·E, while impressive for its time, often fell short of photorealism. The generated images frequently exhibited artifacts, inconsistent lighting, or anatomical inaccuracies when depicting people or animals. For practical applications in design, marketing, or creative industries, the quality gap between generated images and professional photography or illustration remained significant. Users could identify AI-generated content from obvious deficiencies in quality and coherence before considering subtler tells.

Additionally, the original DALL·E's approach struggled with complex compositional requirements. While it could generate images of simple objects or scenes, it had difficulty when prompts required multiple objects, specific spatial relationships, or precise stylistic elements. The autoregressive generation process, designed for sequential text generation, didn't naturally accommodate the two-dimensional, spatially-aware nature of images. This limitation prevented the model from handling many of the complex, multi-object scenes that users wanted to create.

These limitations created a clear research direction: develop a generation approach that could produce higher quality images, maintain better global coherence, and enable editing capabilities. Diffusion models, which had shown promise in unconditional image generation, appeared to be a natural fit, but the challenge lay in effectively integrating textual guidance to ensure generated images matched user intent. The field needed a solution that could combine the semantic understanding of CLIP with the generation capabilities of diffusion models.