DALL·E (2021) › Limitations

Despite its impressive capabilities, DALL·E had several significant limitations that researchers and users quickly recognized. The model sometimes struggled with precise spatial relationships, especially in complex scenes with multiple objects. While it could generate images from prompts like "a small red block on a large green block," it occasionally produced images where objects appeared in incorrect positions or with incorrect relative sizes. These failures reflected the challenge of translating textual spatial descriptions into precise visual arrangements.

The model also had difficulty with text rendering in images. When users requested images containing specific text, such as signs or labels, DALL·E often generated text that was garbled, misspelled, or completely incorrect. This limitation stemmed from the model's training on image-text pairs where text within images was treated as visual patterns rather than semantic content. The discrete VAE's compression process made it difficult for the model to learn the precise details needed for accurate text rendering.

Another limitation was the model's handling of specific numbers and quantities. While DALL·E could understand concepts like "many" or "few," it struggled with precise counts. A prompt asking for "exactly three cats" might generate images with two, four, or five cats. This reflected the probabilistic nature of the model's generation process and the difficulty of enforcing precise numerical constraints in the discrete token space.

The model also had limitations in its understanding of cause and effect relationships, physical constraints, and realistic lighting and shadows. It could generate creative and visually appealing images, but these images sometimes violated basic physics or logic. For example, the model might generate images where shadows point in inconsistent directions or where objects appear to float without support. These limitations highlighted the gap between generating visually coherent images and understanding the underlying physical world.

Additionally, DALL·E's training data contained biases and limitations that could appear in generated images. The model sometimes reproduced stereotypes or generated images that reflected the biases present in its training data. This raised important questions about responsibility in AI systems and the need for careful consideration of training data sources and model behavior.