Multimodal Training Data › Interleaved Documents: Learning from Natural Layouts

While image-text pairs teach basic alignment, they fail to capture how humans process multimodal information in real-world contexts. When you read a Wikipedia article, scientific paper, or instructional manual, images do not arrive as isolated islands with captions. They appear inline, referenced by surrounding text that might say "as shown in Figure 3" or "the red arrow indicates the mechanism." This contextual embedding requires models to maintain coherence across modality boundaries and understand temporal or spatial relationships between text and images.

The limitation of pure paired training becomes clear when you try to explain a complex process using both text and images. Consider an article explaining how a heart valve works: the text describes the opening mechanism, a first image shows the valve open, a second image shows it closed, and a following paragraph explains when each state occurs. A model trained only on individual image-caption pairs has no mechanism for learning that the two images stand for different states of the same structure, or that the text between them gives the causal explanation. It treats each image in isolation rather than as part of a coherent multimodal narrative.

Interleaved documents contain sequences where images and text tokens alternate naturally. A document might look like:

This structure requires the model to perform several advanced cognitive tasks. First, it must maintain long-range dependencies across both modalities, remembering that a statement in paragraph three refers to a diagram shown in paragraph one. Second, it must resolve coreference expressions such as "as shown above," "the red curve," or "this mechanism," mapping linguistic references to specific visual elements. Third, it must understand spatial relationships described in text relative to visual content, such as interpreting "to the left of the nucleus" when viewing a cellular diagram.

Unlike paired data where the relationship is simple and local, interleaved data requires tracking the flow of discourse across multiple images and text segments. The model learns that images serve rhetorical purposes: they illustrate concepts, give evidence, or offer alternative representations of textual information. This understanding is needed for document understanding, scientific reasoning, and following complex instructions that reference visual elements.

The rhetorical role of images in documents is varied and important. Sometimes an image is purely decorative, adding visual interest without informational content. Sometimes it is evidentiary. This gives visual proof of a textual claim. Sometimes it is definitional, showing what a technical term refers to. And sometimes it is sequential, showing steps in a process that would be tedious to describe entirely in text. A model that learns from interleaved documents must develop sensitivity to these different roles, learning from context when an image should be described generically versus when it should be analyzed in detail as evidence for a surrounding claim.

Constructing interleaved datasets involves scraping HTML documents and processing them to extract coherent sequences. The Multimodal C4 (MMC4) and OBELICS datasets exemplify this approach. The pipeline involves several stages of filtering and processing to transform raw HTML into clean training sequences.

The first stage is document selection. Starting with web crawl data from Common Crawl, the system filters for HTML pages containing images embedded within flowing text rather than galleries or separate sections. Pages that merely collect images without narrative context are rejected. A news article with embedded photographs, an educational tutorial with diagrams, and a Wikipedia article with figures all pass this filter. A photo gallery, a product grid page, or a page consisting entirely of a banner image do not.

The second stage is context extraction. For each image, the system extracts the surrounding text window, typically 100 to 200 tokens before and after the image tag, to establish context. This window captures the sentences that introduce, describe, or reference the visual content. The challenge is that web HTML contains many elements that are not real text content: navigation menus, advertisements, cookie consent notices, footer links, and comment sections all appear as text in the DOM but do not constitute real prose. These elements must be stripped before extracting context windows.

The third stage is quality filtering. This removes documents with poor language quality, excessive boilerplate text, or images that appear unrelated to the surrounding context. This step often uses language models to assess coherence and relevance. A paragraph discussing quantum mechanics followed by an image of a cat is likely a layout artifact (perhaps from a sidebar), not real multimodal content. A paragraph discussing animal behavior followed by an image of a cat is real interleaved content. Distinguishing these requires meaning-based matching of both the text and image.

The fourth and final stage is sequence construction. This arranges images and text in document order, creating a linear sequence where special tokens denote image boundaries and maintain the original reading flow. The resulting sequences vary widely in length and image count, from single-image articles to textbooks with dozens of figures per chapter.

Flamingo and similar models process these sequences autoregressively, predicting the next token (whether text or image representation) given the multimodal context. This differs fundamentally from contrastive learning. Instead of learning to match isolated pairs, the model learns to generate coherent continuations across modality boundaries. When the model sees text describing a concept followed by an image token, it learns to predict visual features that would naturally follow that textual introduction. When it sees an image followed by text, it learns to generate descriptive or analytical language appropriate to that visual content.

Interleaved data introduces unique preprocessing complexities that require careful architectural consideration. These challenges stem from the variable and high-dimensional nature of visual content compared to discrete text tokens.

Variable image counts present the first major challenge. Documents contain different numbers of images, ranging from zero to hundreds in long-form content such as textbooks or photo essays. The model must handle variable-length image sequences, typically by inserting special <image> tokens into the text stream and processing visual features at those positions. This requires dynamic computation graphs where the number of visual inputs changes per sample, complicating batching and memory management. While text batches simply pad shorter sequences to the length of the longest, multimodal batches must also handle variable numbers of visual feature tensors, requiring custom collation logic.

Position encoding across modalities requires careful handling. As discussed in Part XIV: Positional Encoding, transformers rely on position information to understand sequence order. In interleaved documents, images occupy positions in the sequence but do not consume text tokens. If a document has text tokens 1 through 50, followed by an image, then text tokens 51 through 100, the image must be assigned a position index that shows its location in the narrative flow without disrupting the text positions. Different architectures handle this differently: some assign the image the position of the subsequent text token, others insert special position markers, and some use separate position encodings for each modality. The choice affects how well the model learns to use positional cues for cross-modal reference resolution.

Memory constraints become severe with long documents containing many high-resolution images. A single document with twenty high-resolution images might require more GPU memory than available, even with gradient checkpointing. Training requires aggressive image resizing, frame sampling for video content, or sparse attention patterns like those covered in Part XVII: Efficient Attention. Some systems process images at lower resolution during pre-training, then increase resolution during fine-tuning to manage this constraint. This two-stage resolution strategy is an engineering compromise: the model learns semantic content from lower-resolution images at scale, then learns fine-grained visual details from higher-resolution images in a smaller but more expensive fine-tuning phase.

Cross-modal attention patterns present a deeper architectural challenge. In pure text transformers, every token can attend to every other token. In interleaved models, it is unclear whether text tokens should attend to image features from earlier in the sequence, image features from later in the sequence, or both. Flamingo uses a Perceiver Resampler architecture that compresses each image's visual features into a fixed number of visual tokens, then uses cross-attention at each transformer layer to allow text tokens to query these visual features. This design gives controlled, efficient access to visual information without modifying the underlying language model's self-attention mechanism, letting pre-trained language models to be extended to visual inputs without catastrophic forgetting of their textual capabilities.