CLIP (2021) › The Problem

Traditional computer vision systems faced fundamental limitations in how they learned and what they could understand. The dominant paradigm for training vision models involved collecting large datasets with fixed label sets, such as ImageNet's one thousand categories or CIFAR's ten classes. Each category required hundreds or thousands of labeled examples for the model to learn effectively. This approach created several problems. First, it was expensive and time-consuming to collect labeled data, particularly for specialized domains or rare categories. Second, models trained this way could only recognize categories they had seen during training, making them inflexible when faced with new tasks or concepts. Third, these systems understood images only in terms of predefined categories, lacking the rich semantic understanding that natural language could provide.

The disconnect between vision systems and natural language created additional challenges. Humans describe images using rich, compositional language that goes far beyond simple category labels. We might describe an image as "a golden retriever playing fetch in a sunny park" or "a vintage car parked on a cobblestone street at sunset." Traditional vision models, constrained to fixed categories, couldn't capture this richness. They might correctly identify "dog" or "car," but they missed compositional descriptions, spatial relationships, and contextual details that natural language conveys. This limitation made it difficult to use vision models in applications where human users wanted to describe what they were looking for in natural terms.

Image retrieval and search applications faced particularly severe limitations. Existing systems typically relied on keyword matching or required extensive labeled training data for specific search queries. A user wanting to find images of "happy people at a beach party" couldn't easily express this to a traditional vision system that only understood fixed categories like "person" and "beach" separately. The system couldn't understand the compositional nature of the query or the semantic relationships between concepts. Similarly, zero-shot learning scenarios, where models needed to recognize categories they hadn't been trained on, were largely impossible with traditional approaches.

Another fundamental problem was the lack of alignment between visual and textual representations. Vision models learned representations optimized for classification accuracy on their training sets, while language models learned representations optimized for text understanding. These representations existed in separate spaces with no natural way to connect them. Attempts to bridge this gap typically required task-specific training that didn't generalize well. Researchers struggled to create systems that could understand, for example, that the text "a red stop sign" should be semantically close to images of red stop signs, even if the model had never explicitly been trained on that association.

The problem was particularly acute for practical applications requiring flexibility and generalization. Real-world vision tasks often involve rare categories, evolving requirements, or novel combinations of concepts that weren't present in training data. A security system might need to recognize new types of objects. A content moderation system might need to understand evolving concepts of inappropriate content. An artistic application might need to retrieve images based on abstract descriptions. Traditional supervised learning approaches, requiring extensive labeled data for each new task or category, couldn't scale to meet these needs.