Multimodal Training Data: Image-Text Pairs & Curation

Multimodal models do not learn to see by reading textbooks. Unlike large language models that can consume trillions of tokens scraped from the web, vision-language models require a fundamentally different data architecture. They must align pixel arrays with linguistic descriptions, learn from documents where images interrupt prose mid-sentence, and follow complex instructions that reference visual details. The quality and structure of this training corpus determine what the model knows and whether it can ground language in visual reality or merely hallucinate correlations.

The core challenge is one of representation translation. A photograph of a dog encodes information as spatial patterns of color intensity across millions of pixels. The word "dog" encodes information as a discrete symbol in a vocabulary of tens of thousands of tokens. These two representations share no obvious structural relationship: you cannot compute the linguistic meaning of "dog" by averaging pixel values any more than you can reconstruct a photograph from its caption. Bridging these fundamentally different representational systems requires carefully constructed training data that gives the right form of supervision at each stage of learning.

This chapter examines the three primary data regimes that power modern multimodal systems. We begin with image-text pairs, the foundation of contrastive pre-training used by models like CLIP to establish basic vision-language alignment. We then explore interleaved documents, the web-scraped HTML and PDF layouts that teach models like Flamingo to reason across long contexts where visual and textual information interleave naturally. Finally, we address visual instruction data, the carefully curated conversations and reasoning chains required to transform a pretrained vision encoder into an assistant that follows complex multimodal directives. Throughout, we confront the necessary challenge of data quality: how to filter noise, balance distributions, and prevent the degenerate shortcuts that cause models to exploit captioning biases rather than learn real visual understanding.

Understanding these data regimes is needed because data determines capabilities alongside architecture. A model trained exclusively on image-text pairs learns to match visual patterns to linguistic concepts but struggles with contextual reasoning that spans multiple images. A model trained on interleaved documents develops richer contextual understanding but may never learn to follow precise instructions unless fine-tuned on conversation data. The three regimes build on each other in a deliberate curriculum, and grasping why each stage is necessary illuminates why modern vision-language models are built the way they are.

The simplest and most scalable form of multimodal supervision pairs a single image with a descriptive caption. This format, seemingly trivial (an image of a dog alongside the word "dog"), encodes a significant learning signal. The model must learn to map high-dimensional visual features into the same semantic space as linguistic concepts. The model needs an abstract representational space where visual semantics and linguistic semantics become interchangeable, letting zero-shot transfer between modalities.

The challenge lies in the basic difference between these modalities. Vision operates on continuous, high-dimensional pixel grids where local correlations matter intensely. Language operates on discrete, sequential tokens where syntactic structure and long-range dependencies dominate. Bridging these requires large amounts of parallel supervision, where the model learns that a specific constellation of edges and colors alongside textures in the visual field corresponds to the abstract concept denoted by a word or phrase.

Consider what the model must learn from a single pair: an image of a golden retriever beside the caption "a dog playing in autumn leaves." The word "dog" must be grounded in the visual pattern of fur, four legs, a muzzle, and ears. "Playing" must be associated with specific postures and motion cues. "Autumn leaves" requires recognizing both the presence of leaves and their brownish-orange color. None of these groundings are given; the model must discover them by processing millions of such pairs and learning that certain visual features consistently co-occur with certain linguistic tokens. This statistical discovery process is the foundation of visual representation learning.

Image-text pair datasets typically derive from three primary sources, each with distinct advantages and contamination risks.

Alt-text scraping crawls the web to extract HTML alt attributes associated with images. These alternative text descriptions were originally intended for accessibility. This gives screen reader descriptions for visually impaired users. Datasets like Conceptual Captions (CC12M/CC3M) and LAION-5B use this approach at large scale, harvesting billions of pairs from the Common Crawl corpus. The advantage is scale and diversity, capturing the long tail of human visual experience across cultures and domains as well as visual styles. The disadvantage is noise, as alt-text often contains SEO keywords, file names, or template text rather than real descriptions. A webmaster optimizing for search rankings might write "buy cheap blue widgets" in an image's alt attribute rather than describing what the image shows, and this corrupted pair would be included in the raw dataset.

Structured metadata uses titles and descriptions alongside tags from photo-sharing platforms such as Flickr, Wikimedia Commons, or e-commerce sites. These sources often give higher quality annotations because users intentionally describe their content for searchability or archival purposes. Wikimedia Commons entries, for instance, often contain detailed descriptions written by volunteers who care about accuracy and completeness. E-commerce sites give structured attributes (color, size, category) alongside descriptions. However, these sources introduce their own biases. Flickr overrepresents photography enthusiasts in wealthy countries. E-commerce sites overrepresent consumer products. Wikimedia overrepresents culturally significant landmarks and events. When models train primarily on these sources, they develop uneven coverage: excellent at recognizing consumer goods and tourist destinations, poor at everyday objects from non-Western contexts.

Synthetic generation uses captioning models or large language models to generate descriptions for unlabeled images. This approach can rapidly scale datasets by applying existing vision-language models to unlabeled image collections. However, this method risks compounding model biases and hallucinations. If the teacher model tends to describe certain demographics in stereotypical ways, or fails to recognize specific visual patterns, these errors propagate into the training data for the next generation of models. Also, synthetic captions often lack the creative variability and cultural nuance of human-written text. A model asked to caption a thousand images of coffee will tend to produce variations on the same description, lacking the linguistic diversity that makes web-scraped captions useful for learning rich language grounding.

The raw web data requires aggressive filtering. As we discussed in Part XX: Data Curation regarding text-only corpora, web-scraped multimodal data contains substantial noise. Common issues include images with missing or irrelevant alt-text, advertisements where the text describes the product being sold rather than the image content, memes with unrelated or ironic captions, and NSFW content that must be excluded for safety reasons. Without filtering, a model trained on raw web data develops confused associations: it might learn that the word "FREE" is associated with promotional banner images, or that "Click here" correlates with graphic icons, patterns that are statistically true in web data but useless for learning visual semantics.

Image-text pairs enable contrastive learning, which we explored in the context of dense retrieval in Part XLIV, Chapter 4: Contrastive Learning for Retrieval. The objective is structurally identical, using natural pairing supervision to learn a joint embedding space.

The intuition behind contrastive learning is elegant. Given a batch of $N$ image-text pairs, the model learns to maximize the similarity between the correct pairs (positive examples) while minimizing similarity between all $N^2 - N$ incorrect pairings (negative examples). This creates a discriminative task where the model must learn to distinguish the true caption for an image from a set of distractors. Because the distractors are drawn from the same batch, they are semantically plausible but visually incorrect, forcing the model to learn fine-grained visual distinctions rather than relying on coarse semantic categories.

Think of it as a large matching game. If a batch contains images of a cat, a dog, and a bird paired with their respective captions, the model must learn to match each image to its caption and not the others. With a batch size of 32,768 (which large-scale CLIP training uses), the model must correctly identify the one correct caption out of 32,768 options for each image, and vice versa. This extraordinarily hard discrimination task forces the model to develop highly specific, discriminative representations.

For a vision encoder $f_\theta$ and text encoder $g_\phi$, the similarity between image $x_i$ and text $t_j$ is:

where:

• $s_{ij}$: the scaled similarity score between image $i$ and text $j$
• $f_\theta(\cdot)$: the vision encoder with parameters $\theta$, mapping an image to an embedding vector
• $x_i$: the $i$-th image in the batch
• $g_\phi(\cdot)$: the text encoder with parameters $\phi$, mapping text to an embedding vector
• $t_j$: the $j$-th text caption in the batch
• $\cdot$: the dot product between embedding vectors, measuring cosine similarity before temperature scaling
• $\tau$: a learnable temperature parameter that scales the logits, controlling the sharpness of the probability distribution (lower values make the distribution more peaked)

The temperature parameter plays a useful role in training dynamics. A high temperature smooths the distribution. This makes the model less confident about distinctions between similar pairs. A low temperature sharpens the distribution, forcing the model to make harder distinctions. Learning this parameter allows the model to automatically calibrate the appropriate level of discrimination for the dataset.

The success of contrastive learning on image-text pairs surprised many researchers when CLIP was introduced in 2021. Supervised learning on carefully labeled datasets had dominated computer vision, and it was unclear whether noisy web-scraped pairs would give sufficient signal. The answer turned out to be: they do, if the batch is large enough.

The key insight is that large batches implicitly construct hard negatives. In a batch of 32,768 pairs, many images share semantic similarity: there might be ten different images of dogs, five images of beaches, and several images of sporting events. When the model must distinguish between "a golden retriever catching a frisbee" and "a labrador running on the beach," it cannot rely on coarse category matching; it must learn the fine-grained visual and linguistic features that distinguish these descriptions.

Also, contrastive learning learns what things are by learning what they are not. The model develops representations where dog images cluster together and separate from cat images, not because it was told to cluster dogs, but because the only way to match captions to images is to develop these discriminative representations. This emergent clustering lets zero-shot classification: a model trained purely on noisy captions can recognize specific object categories it was never explicitly trained to classify, because it learned to place visual and linguistic representations of those categories in the same region of embedding space.

The effectiveness of contrastive pre-training scales with both dataset size and diversity, but the relationship is fine-grained. LAION-5B contains 5.85 billion image-text pairs, yet studies consistently show that filtering for aesthetic quality and text-image alignment (using CLIP similarity scores) often yields better models than using the full noisy corpus. This creates a bootstrapping paradox: we need a good model to filter data to train a better model.

This paradox has practical implications for dataset construction. Initial models trained on unfiltered data develop biases and blind spots. When these models are used to filter subsequent datasets, they may retain familiar patterns and reject novel but valid examples. For instance, an early model trained primarily on photographs might assign low similarity scores to paintings or sketches, causing them to be filtered out of future training sets and reinforcing the initial bias toward photographic imagery.

Data quality manifests through several necessary dimensions:

• Text-image alignment: The caption should describe the image content, rather than merely being co-located with it on the webpage. Web alt-text often contains SEO keywords, file names, or generic placeholders ("image", "photo", "IMG_001") that teach the model spurious correlations. If the model learns that file names predict image content, it fails to develop reliable visual understanding.

• Aesthetic quality: Low-resolution, watermarked, or artistically degraded images impair visual feature learning. The model may learn to recognize watermarks or compression artifacts as primary features rather than the underlying visual content. High-quality datasets often use aesthetic classifiers trained on human preference data to filter for technically competent photography.

• Diversity: The dataset must cover rare concepts and underrepresented visual domains. Image-text pairs follow a long-tail distribution where common objects (cats, cars, celebrities) dominate, while specialized domains (rare bird species, technical diagrams, microscopic imagery) remain underrepresented. Without explicit stratification, models become experts on popular content while failing on niche but important domains.

• Caption informativeness: Short captions like "a person" teach less than long, specific captions like "a woman in traditional Mongolian dress performing at the Naadam festival." Longer, more descriptive captions give richer grounding but are less common in web-scraped data, where brevity is the norm.

The long tail has significant practical consequences. A model trained on raw web data will learn rich, fine-grained representations for cats and cars, but only superficial representations for axolotls and astrolabes. When these rare concepts appear in downstream tasks, the model may simply fail to recognize them or confuse them with visually similar but more common alternatives. Addressing this requires explicit data balancing strategies: oversampling rare concepts, curating specialized datasets for important domains, or using retrieval-augmented approaches that access curated knowledge bases at inference time.

While image-text pairs teach basic alignment, they fail to capture how humans process multimodal information in real-world contexts. When you read a Wikipedia article, scientific paper, or instructional manual, images do not arrive as isolated islands with captions. They appear inline, referenced by surrounding text that might say "as shown in Figure 3" or "the red arrow indicates the mechanism." This contextual embedding requires models to maintain coherence across modality boundaries and understand temporal or spatial relationships between text and images.

The limitation of pure paired training becomes clear when you try to explain a complex process using both text and images. Consider an article explaining how a heart valve works: the text describes the opening mechanism, a first image shows the valve open, a second image shows it closed, and a following paragraph explains when each state occurs. A model trained only on individual image-caption pairs has no mechanism for learning that the two images stand for different states of the same structure, or that the text between them gives the causal explanation. It treats each image in isolation rather than as part of a coherent multimodal narrative.

Interleaved documents contain sequences where images and text tokens alternate naturally. A document might look like:

This structure requires the model to perform several advanced cognitive tasks. First, it must maintain long-range dependencies across both modalities, remembering that a statement in paragraph three refers to a diagram shown in paragraph one. Second, it must resolve coreference expressions such as "as shown above," "the red curve," or "this mechanism," mapping linguistic references to specific visual elements. Third, it must understand spatial relationships described in text relative to visual content, such as interpreting "to the left of the nucleus" when viewing a cellular diagram.

Unlike paired data where the relationship is simple and local, interleaved data requires tracking the flow of discourse across multiple images and text segments. The model learns that images serve rhetorical purposes: they illustrate concepts, give evidence, or offer alternative representations of textual information. This understanding is needed for document understanding, scientific reasoning, and following complex instructions that reference visual elements.

The rhetorical role of images in documents is varied and important. Sometimes an image is purely decorative, adding visual interest without informational content. Sometimes it is evidentiary. This gives visual proof of a textual claim. Sometimes it is definitional, showing what a technical term refers to. And sometimes it is sequential, showing steps in a process that would be tedious to describe entirely in text. A model that learns from interleaved documents must develop sensitivity to these different roles, learning from context when an image should be described generically versus when it should be analyzed in detail as evidence for a surrounding claim.

Constructing interleaved datasets involves scraping HTML documents and processing them to extract coherent sequences. The Multimodal C4 (MMC4) and OBELICS datasets exemplify this approach. The pipeline involves several stages of filtering and processing to transform raw HTML into clean training sequences.

The first stage is document selection. Starting with web crawl data from Common Crawl, the system filters for HTML pages containing images embedded within flowing text rather than galleries or separate sections. Pages that merely collect images without narrative context are rejected. A news article with embedded photographs, an educational tutorial with diagrams, and a Wikipedia article with figures all pass this filter. A photo gallery, a product grid page, or a page consisting entirely of a banner image do not.

The second stage is context extraction. For each image, the system extracts the surrounding text window, typically 100 to 200 tokens before and after the image tag, to establish context. This window captures the sentences that introduce, describe, or reference the visual content. The challenge is that web HTML contains many elements that are not real text content: navigation menus, advertisements, cookie consent notices, footer links, and comment sections all appear as text in the DOM but do not constitute real prose. These elements must be stripped before extracting context windows.

The third stage is quality filtering. This removes documents with poor language quality, excessive boilerplate text, or images that appear unrelated to the surrounding context. This step often uses language models to assess coherence and relevance. A paragraph discussing quantum mechanics followed by an image of a cat is likely a layout artifact (perhaps from a sidebar), not real multimodal content. A paragraph discussing animal behavior followed by an image of a cat is real interleaved content. Distinguishing these requires meaning-based matching of both the text and image.

The fourth and final stage is sequence construction. This arranges images and text in document order, creating a linear sequence where special tokens denote image boundaries and maintain the original reading flow. The resulting sequences vary widely in length and image count, from single-image articles to textbooks with dozens of figures per chapter.

Flamingo and similar models process these sequences autoregressively, predicting the next token (whether text or image representation) given the multimodal context. This differs fundamentally from contrastive learning. Instead of learning to match isolated pairs, the model learns to generate coherent continuations across modality boundaries. When the model sees text describing a concept followed by an image token, it learns to predict visual features that would naturally follow that textual introduction. When it sees an image followed by text, it learns to generate descriptive or analytical language appropriate to that visual content.

Interleaved data introduces unique preprocessing complexities that require careful architectural consideration. These challenges stem from the variable and high-dimensional nature of visual content compared to discrete text tokens.

Variable image counts present the first major challenge. Documents contain different numbers of images, ranging from zero to hundreds in long-form content such as textbooks or photo essays. The model must handle variable-length image sequences, typically by inserting special <image> tokens into the text stream and processing visual features at those positions. This requires dynamic computation graphs where the number of visual inputs changes per sample, complicating batching and memory management. While text batches simply pad shorter sequences to the length of the longest, multimodal batches must also handle variable numbers of visual feature tensors, requiring custom collation logic.

Position encoding across modalities requires careful handling. As discussed in Part XIV: Positional Encoding, transformers rely on position information to understand sequence order. In interleaved documents, images occupy positions in the sequence but do not consume text tokens. If a document has text tokens 1 through 50, followed by an image, then text tokens 51 through 100, the image must be assigned a position index that shows its location in the narrative flow without disrupting the text positions. Different architectures handle this differently: some assign the image the position of the subsequent text token, others insert special position markers, and some use separate position encodings for each modality. The choice affects how well the model learns to use positional cues for cross-modal reference resolution.

Memory constraints become severe with long documents containing many high-resolution images. A single document with twenty high-resolution images might require more GPU memory than available, even with gradient checkpointing. Training requires aggressive image resizing, frame sampling for video content, or sparse attention patterns like those covered in Part XVII: Efficient Attention. Some systems process images at lower resolution during pre-training, then increase resolution during fine-tuning to manage this constraint. This two-stage resolution strategy is an engineering compromise: the model learns semantic content from lower-resolution images at scale, then learns fine-grained visual details from higher-resolution images in a smaller but more expensive fine-tuning phase.

Cross-modal attention patterns present a deeper architectural challenge. In pure text transformers, every token can attend to every other token. In interleaved models, it is unclear whether text tokens should attend to image features from earlier in the sequence, image features from later in the sequence, or both. Flamingo uses a Perceiver Resampler architecture that compresses each image's visual features into a fixed number of visual tokens, then uses cross-attention at each transformer layer to allow text tokens to query these visual features. This design gives controlled, efficient access to visual information without modifying the underlying language model's self-attention mechanism, letting pre-trained language models to be extended to visual inputs without catastrophic forgetting of their textual capabilities.

Pre-training on image-text pairs and interleaved documents yields a model that can stand for visual content and generate needed text. However, like their text-only counterparts discussed in Part XXXVI: Instruction Tuning, these models do not naturally follow instructions or engage in helpful dialogue. They require fine-tuning on visual instruction data: curated examples of multimodal conversations, reasoning chains, and task completions that teach the model to act as a helpful assistant rather than merely a passive describer.

The gap between pre-training and instruction following is particularly pronounced in multimodal settings. A model pre-trained on alt-text learns to describe images, but not to answer questions about them, follow complex directives involving multiple reasoning steps, or engage in back-and-forth dialogue that references specific visual details. If you ask a model trained purely on captioning data "What would happen if I added more water to the mixture shown in the beaker?" it has no framework for this hypothetical, reasoning-intensive query. It can describe the beaker and its contents, but not reason about chemical processes or answer counterfactual questions.

Instruction tuning bridges this gap by exposing the model to the format and style of assistant interactions. The model learns what to generate and how to respond to specific types of requests: questions and instructions alongside comparisons and multi-step reasoning tasks. This move from passive description to active assistance is not automatic; it requires carefully designed training data that covers the full range of tasks users will request.

Visual instruction datasets structure interactions as conversations or task completions, each serving different pedagogical purposes. The format determines what skills the model develops, from simple recognition to complex multimodal reasoning.

Single-turn QA is the simplest format, consisting of an image, a question, and an answer. For example:

• Image: [photo of ingredients on a counter]
• Question: "What can I make with these ingredients?"
• Answer: "With tomatoes, basil, mozzarella, and olive oil, you can make a Caprese salad. This Italian dish highlights fresh ingredients and requires no cooking."

This format teaches the model to extract specific information from images and map it to domain knowledge. It is useful for visual question answering but does not teach conversational coherence or multi-step reasoning. The model learns to process a single image-question pair and generate a needed answer, but has no mechanism for maintaining context across turns or building on prior exchanges.

Multi-turn conversation extends this to dialogue format:

• User: [image of graph] "What trend does this show?"
• Assistant: "The graph shows increasing global temperatures from 1880 to 2020, with particular acceleration in recent decades."
• User: "What caused the sharp rise after 1980?"
• Assistant: "This acceleration correlates with increased industrial CO2 emissions, deforestation, and greenhouse gas concentrations. The slope change around 1980 marks when global industrialization intensified."

This format teaches the model to maintain context across exchanges, resolve pronouns and references, and give information incrementally based on user follow-up questions. It is needed for chatbot applications and interactive assistance. Notice that in the second user turn, there is no image reference, yet the model must remember which graph was discussed and connect the follow-up question to the visual information from earlier in the conversation.

Visual reasoning chains give explicit intermediate steps:

• Image: [geometry problem diagram]
• Question: "Find the value of angle x."
• Reasoning: "First, I notice that lines AB and CD are parallel. Therefore, angle y equals angle z due to alternate interior angles. Since angle x and angle z form a linear pair, they must sum to 180 degrees. Given that angle z is 60 degrees based on the provided measurement..."
• Answer: "120 degrees"

Chain-of-thought formatting teaches the model to decompose complex problems, verbalize its reasoning process, and check intermediate steps before giving final answers. This materially improves performance on mathematical and logical reasoning tasks involving visual inputs. The key insight from research on chain-of-thought prompting, which we explored in Part XXX: Emergent Capabilities, carries over directly to multimodal settings: models that explain their reasoning steps make fewer errors because intermediate steps constrain and check each other.

Grounded instruction following links language to specific visual regions:

• Image: [city street scene]
• Instruction: "Describe the safety hazard visible in the lower-left quadrant."
• Response: "A manhole cover is slightly raised in the lower-left area, creating a tripping hazard for pedestrians. The uneven surface also poses a risk for cyclists and could cause a wheel to catch."

This format teaches spatial localization: the model must understand the image globally and identify specific regions referenced by coordinates, directions, or object identifiers. This capability is needed for assistive technologies, quality inspection systems, and any application requiring precise visual localization.

Creating high-quality visual instruction data is expensive and labor-intensive. Three approaches dominate the field, each with distinct trade-offs between cost and scale as well as quality.

Human annotation has expert annotators write questions and answers for images, often following detailed guidelines to ensure diversity and depth. This yields high quality but limited scale, typically creating hundreds of thousands rather than millions of examples. Datasets like VQAv2 and OK-VQA use this approach. The advantages include high accuracy, creative diversity, and the ability to capture fine-grained cultural contexts. Skilled annotators ask questions that naturally arise when viewing an image, including questions about relationships and comparisons as well as counterfactuals and implied information that automated systems miss. The disadvantages are cost and time, plus potential annotator biases in how questions are framed and answered. Annotators from the same cultural background may share biases about what questions are interesting or what the "obvious" answer is.

GPT-4V assisted generation uses powerful multimodal models to generate instruction data at scale. The LLaVA approach illustrates this pipeline: collect image captions from CC3M or similar datasets, use GPT-4 to generate diverse instructions and conversations alongside reasoning questions based solely on the caption text (without seeing the image), then optionally use GPT-4V to verify that the generated questions are answerable from the image content. This bootstrapping creates millions of training examples at low cost, letting rapid scaling of instruction datasets.

However, this approach inherits the biases and hallucination tendencies of the teacher model. If GPT-4V tends to make certain types of errors or exhibits specific cultural biases, these propagate into the training data. Also, questions generated from captions rather than actual images may ask about details not visible in the image, creating misaligned training signals. A caption might mention "a red car," but the car may be partially occluded in the actual image, making questions about its color unanswerable from the visual content alone. When the student model trains on these examples, it learns to answer confidently about visual details that cannot be seen, encoding hallucination behavior from the start.

Conversion of academic datasets reformats existing computer vision benchmarks (VQA, captioning, object detection, OCR) as instruction-following examples. For instance, a detection annotation becomes an instruction: "Locate all the cars in this image and give their bounding boxes," with the response containing structured coordinates in JSON format. A segmentation mask becomes a color description task. An OCR annotation becomes a text extraction instruction.

This approach uses carefully curated academic datasets with high-quality annotations. The advantage is accuracy and diversity of task types: academic benchmarks were designed to test specific visual capabilities, and converting them to instruction format injects these capabilities into the instruction-tuned model. The disadvantage is that these datasets often contain distributional biases (specific photography styles, object categories, or compositional patterns from benchmark design) and may not stand for natural user queries. They also lack the conversational flow of real human interactions, potentially making the model overly formal or structured in its responses.

Visual instruction data requires careful formatting to use pre-trained text templates. As we covered in Part XXXVI, Chapter 4: Instruction Format, chat templates structure conversations with special tokens that indicate speaker turns and message boundaries.

For multimodal models, templates typically insert image tokens at the start of sequences to establish the visual context before the conversation begins:

The <image> token (often represented as special token IDs or as placeholder tokens replaced by visual features during the forward pass) signals where to inject the image encoder's output into the language model's context. Different architectures handle this injection differently. Some models prepend image features to the text embeddings, treating them as additional context tokens. Others interleave them at specific positions marked by the token.

System prompts may also specify the model's role: "You are a helpful assistant that can see and analyze images. Answer questions accurately and concisely." This framing helps the model transition from pre-training (where it simply described images) to instruction following (where it assists users with specific tasks). The system prompt also helps constrain undesirable behaviors inherited from pre-training: a model that learned from diverse web text might generate inappropriate content unless the system prompt establishes appropriate assistant behavior.

The choice of template format also affects what edge cases the model handles gracefully. If instruction tuning data always places the image at the beginning of the conversation, the model may struggle when images appear mid-conversation or when multiple images appear at different points in a dialogue. The diversity of instruction template formats in the training data determines how reliably the model handles varied real-world usage patterns.

The success of multimodal training depends less on raw scale than on careful curation. Poor data quality manifests as specific, predictable failure modes. Models may fail to recognize uncommon orientations because training images are always upright, or generate stereotypical captions because the dataset contains demographic biases, or hallucinate objects that are commonly mentioned in captions but absent from images. Each of these failure modes traces directly to a specific data quality problem that curation should address.

Quality curation requires systematic evaluation across multiple dimensions, balancing the need for scale against the risk of noise. The challenge is that each dimension of quality involves trade-offs: improving aesthetic quality may reduce cultural diversity; improving text-image alignment may favor simple, obvious captions over fine-grained descriptions; improving content safety may inadvertently filter legitimate educational or documentary content.

Aesthetic and technical quality assessment filters low-quality images that impair feature learning by teaching the model to recognize artifacts rather than content. Automated filters assess several properties. Resolution and aspect ratio filters remove tiny images or those with extreme aspect ratios that would be distorted during resizing. Compression artifact detection identifies blocky regions or mosquito noise around edges that indicate heavy JPEG compression. Watermark detection uses specialized classifiers trained to detect stock photo overlays, as watermarked images often indicate placeholder or preview content without full visual information. Aesthetic scoring, trained on human preference data, predicts whether an image shows competent photography, though this risks encoding subjective cultural standards of beauty that systematically undervalue certain styles or genres.

The tension in aesthetic filtering is that beauty standards are culturally specific. A model trained on an aesthetic classifier that favors sharp focus, good lighting, and rule-of-thirds composition will filter out intentionally blurry photographs used for artistic effect, overexposed images used to create specific moods, and unconventionally composed photographs from photographic traditions that prioritize different values. This risks encoding a narrow aesthetic vocabulary while training the model, limiting its ability to understand the full range of visual expression.

Content safety removes NSFW and toxic content that is necessary for deployment safety. This employs classifiers trained on labeled unsafe content categories (adult imagery, violence, hate symbols), text-based filtering of captions containing slurs, adult terminology, or toxic language, and face detection and blurring for privacy protection. However, safety filtering introduces its own complexities. Overly aggressive safety filters may remove legitimate educational content (medical imagery, historical documentation of violence) along with harmful content. The threshold between "educational" and "harmful" is context-dependent and cannot be resolved by content alone.

Text-image alignment ensures captions describe the image content rather than being incidental text from the webpage. Methods include CLIP similarity scores (filtering pairs with cosine similarity below a threshold, typically 0.2 to 0.3), caption length heuristics (removing "image.jpg," single-word captions, or file extensions), and language detection (removing non-English or multilingual noise for English-centric models, though this perpetuates English-language dominance in AI systems). Each of these heuristics has failure modes: CLIP similarity filters may remove valid captions for visual concepts underrepresented in CLIP's own training; caption length filters may remove legitimately brief but accurate captions; language filters remove non-English content that would be useful for multilingual models.

Duplicate images and near-duplicates (resized, cropped, or filtered versions of the same source) create subtle overfitting and inflate performance metrics artificially. If a test image appears in training, the model may memorize the answer rather than learning to generalize. This is not a theoretical concern: large web-scraped datasets regularly contain thousands of copies of the same popular photographs, viral images, and stock photos that appear across many websites. Training on these duplicates wastes compute and creates false confidence in the model's breadth of experience.

Exact deduplication uses perceptual hashing (pHash), which generates fingerprints reliable to minor modifications such as resizing, re-encoding, or slight cropping. Two images with identical pHash values are almost certainly the same image, even if their file sizes or metadata differ. This catches obvious duplicates at low computational cost.

Approximate deduplication employs more advanced methods. CLIP embedding clustering groups semantically similar images by projecting them into the same embedding space and identifying tight clusters. URL deduplication removes cases where the same image is hosted at multiple URLs, which is extremely common on the web as images propagate through social media shares and content aggregators. Near-duplicate detection uses vision model features to find visually similar but not identical images, catching cases like the same photograph with different color grading or slight crops.

Deduplication is particularly important for evaluation integrity. If benchmark images appear in the training set, the model memorizes answers rather than generalizing. Unlike text, where n-gram overlap detection can identify contamination, image deduplication requires perceptual hashing that misses semantic duplicates: the same scene photographed from slightly different angles. A human photographing the Eiffel Tower from slightly different positions on two different days produces two non-duplicate images that stand for the same visual scene. If one appears in training and the other in evaluation, the model may use memorized associations rather than real visual understanding. This kind of subtle contamination is nearly impossible to fully eliminate at scale.

Raw web data exhibits extreme distributional bias that can severely limit model capabilities. Object bias means that certain objects (phones, cats, cars) appear orders of magnitude more frequently than others (axolotls, astrolabes, specific scientific instruments). Style bias means that photographs dominate over paintings, diagrams, medical imagery, and synthetic renders. Demographic bias means that Western and urban subjects, along with youthful ones, are overrepresented, while elderly individuals, non-Western contexts, and rural environments are underrepresented.

Stratified sampling attempts to balance these distributions by explicitly sampling from underrepresented categories. For object categories, this might mean capping the number of examples per category and oversampling rare categories to reach a more uniform distribution. For visual styles, this might mean explicitly including painting and diagram datasets alongside medical imaging and photographic sources. For demographic coverage, this might mean curating datasets from specific geographic regions or cultural contexts.

However, perfect balance is impossible without manual curation, and aggressive balancing may discard high-quality data from overrepresented categories. A model trained on a perfectly balanced dataset where each object category appears exactly once is less capable on common objects (because it sees each only once) and better on rare objects (because it sees each relative to its original frequency). The optimal balance depends on the target application: a general assistant needs broad but uneven coverage, while a specialized medical imaging tool might deliberately oversample pathological imagery even though it is rare in the general population.

Some datasets employ a "quality versus diversity" trade-off, maintaining high-quality subsets for pre-training convergence while adding diverse tail examples for resilience to rare inputs. The intuition is that the model needs clean, reliable examples to establish strong representations early in training, but needs exposure to diverse edge cases to generalize well at deployment time. This staged approach mirrors curriculum learning strategies from text-only pre-training, where models often train on progressively more challenging or diverse data as they mature.

The highest-quality image-text pairs are often those filtered by existing strong models (CLIP, GPT-4V). This creates a feedback loop where early model biases perpetuate in subsequent training data. If CLIP associates "doctor" with images of men due to training data biases, filtering with CLIP scores may inadvertently retain gender-biased examples and filter counter-stereotypical ones (such as images of female doctors).

This bootstrap problem is insidious because it is difficult to detect. The filtered dataset appears high-quality by automated metrics, yet systematically excludes valid but counter-stereotypical examples. Breaking this cycle requires human oversight, demographic diversity in validation sets, and careful auditing of filtering decisions to ensure they do not reinforce existing biases. Some research groups maintain "bias auditing" datasets: carefully curated test sets designed to reveal demographic biases in model outputs, which can be used to detect when filtering has amplified problematic associations.

Modern multimodal datasets are not filtered once and then frozen; they are maintained as living pipelines that evolve as better filtering tools and quality signals become available. DataComp, an academic benchmark for multimodal data curation, allows researchers to compare different curation strategies on a fixed pool of candidate pairs, isolating the effect of curation from the effect of model architecture. This kind of systematic comparison has revealed important insights: simple heuristics like filtering for captions with at least five words and images with resolutions above 200 pixels often outperform complex learned filters, suggesting that basic quality signals are more reliable than model-dependent quality assessments.

The filtering threshold for CLIP similarity is a basic policy decision. Setting the threshold at 0.2 retains many noisy pairs but ensures broad coverage of visual domains. Setting it at 0.3 removes more noise but also removes many valid pairs involving visual concepts that CLIP itself is poorly. Operators typically tune this threshold on small validation sets, measuring downstream model performance on targeted benchmarks for specific capabilities rather than optimizing a single aggregate metric.

Let us examine how to load and filter before batching multimodal training data using modern tools. We will use the HuggingFace datasets library for handling large-scale image-text collections and show preprocessing pipelines needed for training vision-language models.

The filtering curve reveals the basic quality-quantity trade-off in multimodal dataset curation. At low thresholds (0.10), nearly all data is retained but quality is low. At high thresholds (0.40), quality is high but most data is discarded. The optimal threshold balances these competing goals, and its location depends on the specific capabilities being optimized. Models trained on larger, noisier datasets sometimes generalize better to rare visual domains, while models trained on smaller, cleaner datasets perform better on benchmarks that reflect the clean distribution.

The construction of multimodal training data faces basic tensions between scale and quality, automation and accuracy, that shape model capabilities and limitations in significant ways. These tensions are not merely technical inconveniences; they determine whether models develop real understanding or mere statistical correlations, whether they serve diverse global populations or only wealthy Western users, and whether they can be trusted for necessary applications.

Image-text pairs teach correlation, not causation or physical understanding. A model trained on "person holding umbrella" captions learns to associate umbrellas with rain, but not that umbrellas cause dryness or that rain falls downward. This produces models that excel at retrieval and captioning yet struggle with physical reasoning or counterfactual scenarios ("what if the umbrella were inside out?").

Interleaved data mitigates this by giving contextual reasoning chains, but the underlying supervision remains correlational. The model learns that certain visual patterns co-occur with certain linguistic patterns, not the physical, causal, or social relationships that connect them. This limitation becomes apparent when models confidently describe impossible scenes or fail to recognize physical contradictions in generated images. A model asked whether a photograph shows physically plausible bridge construction may hallucinate an affirmative answer if the image superficially resembles training examples where such captions were common.

The gap between correlation and causation is not unique to multimodal models: it affects all statistical learning systems. What makes multimodal training data particularly vulnerable is the diversity of ways that text and images can be spuriously correlated on the web. Photographs of luxury goods often appear with aspirational captions that do not describe the image but rather the desired emotional response. News photographs often appear with captions that describe context invisible in the image. Art photographs often appear with abstract descriptions that resist literal visual interpretation. Each of these creates correlations that the model must somehow learn to discount.

The large majority of high-quality image-text pairs feature English captions. While Part XXV: Encoder-Decoder Models discussed multilingual training for text-only models, multimodal models face a steeper challenge. Visual concepts may be culture-specific, such as different hand gestures, architectural styles, or food presentations. Non-English captions are often lower quality or direct translations that lose culturally-specific visual nuance.

This limits transfer to low-resource languages and embeds Western visual assumptions as universal defaults. A model trained primarily on English captions may fail to recognize culturally specific practices, clothing, or environments, effectively rendering invisible the visual experiences of billions of people. Even when non-English text is included, the association between visual and linguistic representations may be weaker for non-English languages because fewer training examples reinforce those connections. The model may default to English-language descriptions when prompted in other languages, even when the image contains culturally specific content that has natural names in the target language.

Addressing this requires intentional curation of non-English, non-Western visual content with high-quality native language captions. Some research groups have created targeted multilingual datasets that explicitly pair images from specific cultural contexts with captions written by native speakers rather than translated from English. These efforts are promising but face the basic challenge of scale: creating billions of high-quality multilingual image-text pairs requires an enormous sustained human effort.

Vision-language models memorize training examples. When benchmark images appear in pre-training data, models reach inflated performance by retrieval rather than reasoning. Unlike text, where n-gram overlap detection can identify contamination, image deduplication requires perceptual hashing that misses semantic duplicates (the same scene photographed from slightly different angles).

As models train on increasingly large web-scraped datasets, the probability that evaluation benchmarks are "leaked" into training approaches certainty, complicating the measurement of true generalization that we will explore in upcoming chapters on evaluation. This is particularly concerning for safety-necessary applications where we need to know whether the model understands concepts or merely recognizes specific test instances.

The memorization problem also raises privacy concerns. Models trained on web-scraped data may memorize specific individuals' photographs, letting identification of private individuals from generated or described content. This is analogous to the text memorization problem, where language models can reproduce training text verbatim, but with the added concern that visual memorization involves personal images that individuals may not have consented to include in training data.

Aggressive filtering improves model quality but reduces diversity and scale. LAION-5B filtered to high-aesthetic, high-alignment subsets retains only 10-20% of original data. This creates a tension: Chinchilla scaling laws from Part XXIII: Scaling Laws suggest models need large data, yet quality constraints limit available quantity.

The result is that compute-optimal training often becomes data-constrained for multimodal models, driving interest in synthetic data generation despite its compounding bias risks. Researchers must choose between training on large amounts of noisy data (risking learning spurious correlations) or small amounts of clean data (risking poor generalization to rare concepts). Neither option is clearly superior: the right balance depends on the target application, the available compute budget, and the specific failure modes that are most consequential.

One promising direction is staged training with different data mixtures at different stages. Early training uses large, noisy datasets to establish broad visual and linguistic coverage. Later training uses smaller, higher-quality datasets to refine representations and reduce noise-induced biases. This mirrors the two-stage approach used in text pre-training, where broad pre-training is followed by targeted domain-specific fine-tuning.

Training data shows and amplifies societal biases. Image-text pairs associate certain professions with specific genders or ethnicities due to photography distribution on the web. Stock photo sites, which contribute heavily to web-scraped datasets, often perpetuate stereotypical representations. When these models generate captions or answer questions about images, they perpetuate these stereotypes, potentially causing real-world harm by reinforcing biased associations.

Unlike text-only models where bias can be mitigated through careful corpus curation or RLHF as covered in Part XXXVII: Alignment and RLHF, multimodal biases are harder to detect because they require visual inspection of the training data, which is computationally expensive at billion-image scale. Auditing for demographic bias requires processing images through detection pipelines that may themselves be biased, creating a methodological challenge for fairness research.

The practical implication is that multimodal models should not be treated as neutral visual analysts. They bring the biases of their training data to every analysis, and users who rely on them for consequential decisions (medical diagnosis, security screening, recruitment) may inadvertently inherit and amplify those biases. Responsible deployment requires both technical mitigation strategies and organizational awareness that model outputs are shaped by data construction choices made during the training pipeline.

Multimodal training data exists in three distinct regimes, each serving different learning objectives and addressing different aspects of vision-language understanding.

Image-text pairs give the scalable foundation for contrastive learning, teaching models to align visual and linguistic representations through millions of simple associations. The contrastive InfoNCE objective treats batch members as negative examples, forcing the model to develop discriminative representations that distinguish similar but distinct visual concepts. The quality of this data depends on aggressive filtering for alignment and aesthetics alongside safety, often creating bootstrapped dependencies on existing strong models. These pairs establish the basic semantic correspondence necessary for zero-shot recognition and retrieval.

Interleaved documents capture natural multimodal reasoning by preserving the context in which images and text co-occur in web documents. This format teaches models to resolve references, maintain long-range dependencies across modalities, and generate coherent narratives that weave between visual and linguistic content. By processing documents in their natural layout, models learn the rhetorical and structural patterns of human multimodal communication. The architectural challenges this data format introduces, including variable image counts, cross-modal position encoding, and memory-efficient long-sequence processing, have driven significant innovations in multimodal model design.

Visual instruction data uses curated multimodal dialogues and reasoning examples to adapt pretrained models for assistant-style behavior. Created through expensive human annotation, model-assisted generation, or benchmark conversion, this data requires careful formatting to fit into language model chat templates and special token protocols. Chain-of-thought examples are particularly useful for building reasoning capabilities, as they teach the model to produce answers and explain the visual and logical steps that lead to them. These examples connect representation learning to assistant behavior.

The curation of these datasets involves constant trade-offs between scale and quality, diversity and safety, automation and accuracy. The filtering decisions made during data preparation, including what CLIP threshold to use, how many images per sequence to allow, and which languages to include, fundamentally constrain model capabilities and biases. As multimodal models grow larger and training becomes increasingly data-constrained, the sophistication of data curation pipelines has become as important as architectural innovation in determining final model performance.

The basic insight is that multimodal training data is not neutral raw material. Each curation decision encodes assumptions about what visual knowledge is important, what quality looks like, and whose visual experience matters. These decisions determine technical performance metrics as well as which communities and domains the model serves reliably and which it handles poorly or harmfully. The future of multimodal AI depends on algorithmic advances and on our ability to construct training corpora that are large yet precise, diverse yet safe, and representative of the full breadth of human visual and linguistic experience.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about multimodal training data.