Flamingo Architecture

Flamingo shifts how we approach multimodal AI. This shows that large language models can acquire visual understanding without catastrophic forgetting of their linguistic capabilities. Rather than training a vision-language model from scratch, Flamingo inserts lightweight trainable components between a frozen vision encoder and a frozen language model. This makes possible the resulting system to process arbitrarily interleaved sequences of text and images while maintaining the ability to perform few-shot learning across vision-language tasks.

This approach differs from earlier methods that typically required joint pretraining of vision and language components from random initialization. By preserving the pretrained weights of both modalities, Flamingo uses the large knowledge already encoded in state-of-the-art vision encoders and language models, avoiding the immense data and computational requirements of training billion-parameter models from scratch. The resulting system exhibits versatility, capable of engaging in visual question answering, image captioning, and multimodal dialogue while retaining the fluency and reasoning abilities of its base language model.

The architecture addresses a core challenge in multimodal AI: how to add new sensory modalities to a capable language model without destroying its existing knowledge or requiring prohibitively expensive end-to-end training. This tension arises because language models develop highly specific internal representations through extensive pretraining on text. When new input modalities are introduced, the distribution of activations can shift sharply, causing the model to forget previously learned linguistic patterns, a phenomenon known as catastrophic forgetting. Flamingo's solution relies on two key innovations: the Perceiver Resampler, which compresses high-dimensional visual features into a fixed number of tokens, and Gated Cross-Attention Dense (GATED XATTN-DENSE) layers, which condition the language model on visual information through carefully gated cross-attention mechanisms. These components are trained while the underlying vision and language models remain frozen, creating an efficient adaptation paradigm that has influenced subsequent vision-language models like OpenFlamingo and IDEFICS.

Before diving into Flamingo's mechanics, let's understand why simply concatenating vision and language models proves insufficient. When we discussed cross-attention in Part XVI, Ch 4: Cross-Attention, we explored how transformers can attend across different modalities. However, standard cross-attention assumes both modalities use compatible representations. Vision encoders typically output feature maps with thousands of tokens (for example, 7x7 or 14x14 grids with high-dimensional features), while language models expect compact token sequences. Directly feeding thousands of visual tokens into every layer of a language model creates computational bottlenecks and disrupts the model's linguistic priors.

The computational challenge is severe: if a vision encoder produces 1,000 tokens per image and we process sequences with multiple images, the attention complexity grows quadratically with the total sequence length. More insidiously, the statistical properties of vision features differ fundamentally from text embeddings. Vision encoders capture spatial hierarchies of edges and textures alongside objects, while language models process sequential symbolic representations. Without careful alignment mechanisms, forcing these incompatible representations to interact can corrupt the carefully tuned feature spaces of either modality.

Consider what happens at the activation level. A pretrained language model has spent hundreds of billions of training tokens learning to produce specific ranges of intermediate activations. The mean and variance of hidden states at each layer have been implicitly optimized so that layer normalizations, feed-forward networks, and attention mechanisms work harmoniously. When you inject visual features with completely different statistical properties into this carefully calibrated system, the layer normalizations begin operating in unexpected regimes, the feed-forward networks receive inputs outside their effective domain, and the attention patterns learned during text pretraining break down. The model does not just fail to understand images; it forgets how to process language.

Training a vision-language model from scratch also requires enormous datasets and computational resources. As we saw in Part XXIII: Scaling Laws, model performance depends heavily on scale, yet retraining billion-parameter models for every new modality combination is economically impractical. The financial and environmental costs of pretraining large transformers from scratch can reach millions of dollars per run. Flamingo asks: can we use existing, capable vision encoders and language models while teaching them to communicate effectively?

Several obvious approaches fail in instructive ways. The simplest idea is to just prepend image tokens to the text sequence and let the language model's self-attention handle everything. This approach has two fatal flaws. First, it requires the language model's input embedding layer to accept a new token type, which means either modifying the pretrained embedding weights (risking forgetting) or projecting vision features into the text embedding space, which gives no mechanism for the model to distinguish visual from textual tokens. Second, the number of image tokens can be enormous (hundreds to thousands for high-resolution images), and forcing the language model to attend to all of them via self-attention at every layer sharply increases computational cost without any architectural safeguard against catastrophic forgetting.

Another approach is to fine-tune the entire model end-to-end on multimodal data, in effect treating vision-language adaptation as standard transfer learning. This approach works when you have enough multimodal data and computational budget, but it is expensive and wasteful: you're spending enormous resources to adjust parameters that were already optimized and shouldn't need to change. More critically, fine-tuning at scale tends to overwrite the fine-grained linguistic capabilities built up during text pretraining, particularly for low-frequency vocabulary, complex reasoning chains, and stylistic register that are underrepresented in typical image-caption datasets.

A third approach uses adapters placed inside the language model, similar to what we explored in Part XXXV: Parameter-Efficient Fine-Tuning. These approaches partially address the parameter efficiency problem, but standard adapters don't naturally handle the sequential, multi-image structure of visual inputs, and they don't address the variable-length output from vision encoders. Flamingo's contribution is to combine the insights from adapter research with a purpose-built compression mechanism and a carefully designed cross-modal attention architecture.

The answer lies in adapter-based architectures that introduce trainable bottlenecks between frozen components. By treating the vision and language models as fixed feature extractors and learning only the interface between them, we minimize the risk of catastrophic forgetting while maximizing data efficiency.

Flamingo consists of three main components arranged in a specific processing pipeline that turns raw pixels and text tokens into coherent multimodal representations:

1. Frozen Vision Encoder: A Normalizer-Free ResNet (NFNet) that extracts high-resolution visual features from input images. This encoder processes images through deep convolutional layers to produce rich spatial feature maps capturing visual semantics at multiple scales.

2. Perceiver Resampler: A trainable module that compresses variable-length vision features into a fixed number of visual tokens (typically 64). This compression is needed for computational tractability, letting the model to process high-resolution images without overwhelming the language model's attention mechanisms.

3. Frozen Language Model with GATED XATTN-DENSE: A pretrained Chinchilla language model augmented with trainable cross-attention layers inserted periodically throughout its depth. These inserted layers act as information channels, letting the language model to reference visual content without modifying its base parameters.

The architecture processes interleaved text and images by treating visual tokens as additional context that the language model can attend to through cross-attention, while the causal masking of the language model ensures autoregressive text generation remains intact. This design choice preserves the language model's ability to generate coherent, fluent text while grounding that generation in visual evidence.

The number of trainable parameters is surprisingly small relative to the total model size. In the largest Flamingo variant (80B parameters), the frozen vision encoder accounts for roughly 435M parameters, the frozen language model accounts for about 70B, and the trainable Perceiver Resampler plus GATED XATTN-DENSE layers account for only about 10B parameters. This means that roughly 87% of the model's parameters are never updated during multimodal training, which explains why the linguistic capabilities are so well preserved.

Vision encoders like NFNet output spatial feature maps representing thousands of local image regions. For a typical 224x224 image processed by an NFNet, the final convolutional layer might produce a 7x7 grid with 2,048-dimensional features, yielding 49 tokens. However, higher resolution images or different architectures can produce thousands of tokens. Feeding all of these into the language model at every layer would be computationally prohibitive, as the cross-attention mechanism would need to compute attention scores between every text token and every visual feature, resulting in memory requirements that scale with the product of text length and visual tokens.

The Perceiver Resampler solves this by learning a fixed number of latent queries that attend to the vision features, compressing the variable-length image representation into a constant number of visual tokens regardless of input resolution. This approach draws inspiration from the Perceiver architecture, which uses a similar mechanism to handle arbitrary input modalities by compressing them into a fixed-size latent bottleneck.

The mechanism works through three stages:

Learned Latents: The resampler maintains $N$ learned query vectors $Q \in \mathbb{R}^{N \times d}$, where $N$ is the desired number of output visual tokens (typically 64) and $d$ is the model dimension. Unlike positional encodings or static embeddings, these latents are optimized during training to extract the most salient visual information for language understanding. You can think of them as $N$ different "questions" the model asks about each image, where the questions themselves are learned rather than hand-designed.

Cross-Attention to Vision: These queries attend to the vision encoder outputs, which serve as keys $K$ and values $V$. If the vision encoder produces $M$ spatial features (where $M$ varies by image size), we have $K, V \in \mathbb{R}^{M \times d}$. The cross-attention operation effectively asks: given these $N$ learned queries, what information from the $M$ vision features is most relevant?

Self-Attention Among Latents: After the initial cross-attention, the resampler applies several layers of self-attention among the $N$ latent vectors, letting them to share information and produce a coherent compressed representation. This step ensures that the final visual tokens capture global image context rather than isolated local features, letting the tokens to collectively stand for the entire image content. Without this self-attention step, each latent would independently summarize the image from its own learned perspective, but the latents would not be aware of each other's "questions" or answers, potentially leading to redundancy or gaps in coverage.

Mathematically, for a single resampler layer, the output latents are computed by first applying cross-attention to the vision features, then applying self-attention among the latents:

where:

• $\text{Latents}_{\text{in}}$: the learned query vectors $Q \in \mathbb{R}^{N \times d}$ before processing the current image
• $\text{VisionFeatures}$: the spatial features output by the vision encoder, serving as keys $K$ and values $V$
• $\text{CrossAttn}$: cross-attention operation where latents attend to vision features
• $\text{SelfAttn}$: self-attention operation among the $N$ latent vectors to share information
• $\text{Latents}_{\text{out}}$: the refined visual tokens after processing

This formulation compresses variable-length image features into a fixed number of tokens regardless of input resolution. The compression ratio can be substantial: for a vision transformer creating 1,024 tokens, compressing to 64 latents yields a 16-fold reduction in computation for downstream cross-attention operations.

The cross-attention operation follows the standard scaled dot-product formulation we covered in Part XIII, Ch 3: Scaled Dot-Product Attention:

where:

• $Q$: the learned latent queries $\in \mathbb{R}^{N \times d}$ (the fixed set of queries attending to image features)
• $K$: keys derived from the vision encoder outputs $\in \mathbb{R}^{M \times d}$
• $V$: values derived from the vision encoder outputs $\in \mathbb{R}^{M \times d}$
• $d_k$: dimension of the keys (used for scaling to prevent softmax saturation)
• $\text{softmax}$: normalization function converting attention scores into a probability distribution over image locations

The scaling factor $\sqrt{d_k}$ prevents the dot products from becoming too large as dimensionality increases, which would push the softmax into regions with extremely small gradients. Without this scaling, the attention distribution would become extremely sharp, effectively attending to only one location and losing the soft, distributed attention that lets rich feature aggregation. The output is a fixed set of $N$ visual tokens that stand for the entire image, regardless of how many patches or regions the vision encoder initially produced.

One subtle but important aspect of the resampler is that the learned latent queries $Q$ are shared across all images processed by the model. This means the resampler applies the same "questions" to every image and learns to extract generally useful visual information rather than queries specialized to particular image categories. The power of the approach comes from the combination of these fixed learned questions with the flexible attention mechanism that routes information from different spatial locations depending on each specific image's content.

The resampler also handles the visual backbone's position encodings. The vision encoder may output features with 2D spatial structure (from convolutional layers) or with 1D position embeddings (from vision transformers). The resampler accepts either format, since it uses cross-attention rather than position-sensitive operations to aggregate information. This architectural flexibility allows Flamingo to use different vision backends without changing the downstream components.

While the Perceiver Resampler handles the vision-to-language dimensionality mismatch, we still need a mechanism to inject visual information into the language model. This must be done without destroying its pretrained linguistic capabilities. Simply adding visual tokens to the input sequence and using self-attention would work, but this approach would require modifying the language model's input embeddings and would not give the fine-grained control needed to prevent catastrophic forgetting. Flamingo adopts a more surgical approach: inserting cross-attention layers directly into the frozen language model at specific intervals.

These aren't standard cross-attention layers, however. They're wrapped in a advanced gating mechanism that ensures training stability and preserves the language model's initial behavior at the start of training. This gating acts as a safety valve, letting the model to gradually learn to use visual information rather than being forced to incorporate potentially noisy gradients from untrained cross-attention weights immediately.

Each GATED XATTN-DENSE block contains:

1. Layer Normalization applied to the input hidden states, stabilizing the distribution of activations before cross-modal interaction
2. Cross-Attention where text tokens (as queries) attend to visual tokens (as keys and values), creating the pathway for visual information to influence text generation
3. A gated residual connection with learnable scalar $\alpha$, controlling how much of the cross-attention output is added to the residual stream
4. Layer Normalization on the output of the cross-attention, preparing the mixed representations for further processing
5. A Dense (FFN) layer processing the mixed representations, letting nonlinear transformation of the visual-textual features
6. Another gated residual connection, independently controlling the contribution of the feed-forward transformation

The gating mechanism is important for preserving the frozen language model's behavior during initial training. Let $x$ be the input to the layer and $F(x)$ be the output of either the cross-attention or dense sublayer. The gated residual connection computes:

where:

• $x$: the input hidden states from the previous layer
• $F(x)$: the output of the cross-attention or dense sublayer after layer normalization
• $\alpha$: a learnable scalar parameter initialized to $0$
• $\tanh(\alpha)$: the gating factor that smoothly interpolates between identity ($0$) and full transformation ($\approx 1$)
• $y$: the output hidden states passed to the next layer

Since $\tanh(0) = 0$, at the beginning of training the layer computes $y = x$, effectively passing the input through unchanged. This initialization strategy acts as a warm start. This keeps the language model doesn't immediately forget its linguistic priors when visual inputs are introduced. As training progresses, $\alpha$ can grow (positively or negatively). This allows the layer to gradually learn to incorporate visual information without causing sudden distribution shifts in the frozen language model's activations.

The choice of $\tanh$ as the gating function deserves attention. A naive approach might use a linear gate, where $y = x + \alpha \cdot F(x)$. But a linear gate has no inherent bound, meaning $\alpha$ could grow unboundedly large during training, eventually overwhelming the residual stream and destroying the frozen model's behavior. Using $\tanh(\alpha)$ bounds the maximum gate value at 1.0. This keeps even with a very large $\alpha$, the cross-attention contribution never exceeds the magnitude of the identity path. This bounded behavior prevents training instability while still letting the model to learn strong visual conditioning when beneficial.

The GATED XATTN-DENSE also includes a separate feed-forward network with its own gating parameter. This FFN is not a standard transformer FFN; it doesn't process text features in isolation but rather processes the mixture of text and visual information that emerges from the cross-attention step. You can think of it as a learned transformation that converts the raw fusion of visual and textual features into a representation better aligned with the language model's expected input distribution. The independent gating of the FFN gives the model flexibility to learn that, for some tasks, simply injecting visual features through cross-attention is sufficient, while for others, a nonlinear transformation of the fused representation is necessary.

Flamingo doesn't add GATED XATTN-DENSE blocks after every transformer layer, as that would be computationally expensive and potentially redundant. Inserting cross-attention at every layer would increase the computational cost proportionally to the depth of the network, and empirical evidence suggests that visual information injected at early layers can propagate effectively through subsequent self-attention layers. Instead, it inserts them at regular intervals, typically every 4th or 6th layer in a 32-layer Chinchilla model. This sparse insertion pattern balances computational efficiency with the need for deep visual conditioning throughout the network. This keeps both low-level visual features and high-level semantic concepts can influence the language model's representations at appropriate levels of abstraction.

The specific choice of insertion frequency involves a tradeoff. Inserting cross-attention after every layer maximizes the model's ability to integrate visual information at fine-grained intervals but roughly doubles the number of cross-attention operations. Inserting every 8 layers reduces overhead materially but may limit the model's ability to tightly couple visual and textual reasoning across all levels of abstraction. The every-4th-layer configuration used in Flamingo was found empirically to strike a good balance.

When processing an interleaved sequence like:

The model processes text tokens autoregressively while maintaining cross-attention to the visual tokens extracted from each image. Importantly, each image's visual tokens attend only to that specific image, while text tokens can attend to all preceding images through the cross-attention mechanism. This architectural choice ensures that visual processing remains image-specific in the resampler, while the language model can freely associate and compare information across multiple images, letting capabilities like visual comparison, temporal reasoning, and multi-image context understanding.

The cross-attention masking implements a specific temporal causality rule: when generating a text token at position $t$, the GATED XATTN-DENSE layers allow attention only to images that appear before position $t$ in the interleaved sequence. This ensures that the model cannot "cheat" by attending to future images that haven't been described yet, maintaining the autoregressive generation property that makes the model useful in practice.

One of Flamingo's most remarkable capabilities is few-shot in-context learning for vision-language tasks, analogous to the text-based in-context learning we explored in Part XXVIII, Ch 4: In-Context Learning. Just as GPT-3 can learn new text tasks from a few examples in the prompt, Flamingo can adapt to new vision tasks by observing a few image-text pairs, without any gradient updates or parameter modifications.

This capability emerges from training on interleaved image-text data from web pages. Unlike paired datasets where one image corresponds to one caption, web documents naturally contain sequences of images interspersed with explanatory text. This natural formatting mirrors the few-shot prompting paradigm, where examples are provided in context followed by a query. Flamingo is trained on the Multimodal MassiveWeb (M3W) dataset, which extracts such interleaved sequences from HTML documents, preserving the natural flow of visual and textual information as it appears in real-world documents.

During training, the model sees sequences like:

The training objective remains standard next-token prediction on the text, but the model must attend to the relevant images through cross-attention to predict tokens accurately. For instance, to predict the word "cat," the model must associate the linguistic token with the visual features of the first image. To predict "Both animals," the model must integrate information from both images, learning cross-image referencing and coreference resolution in the visual domain. This teaches Flamingo to ground linguistic concepts in visual features and to maintain coherence across multiple images.

For few-shot evaluation, we give examples in the same interleaved format:

The model learns to recognize the pattern (image to label) and applies it to the query image, effectively performing few-shot classification without any gradient updates or task-specific fine-tuning. This capability extends beyond simple classification to complex visual reasoning tasks, including visual question answering, optical character recognition, and commonsense reasoning about visual scenes, all from just a handful of examples provided in the context window.

What makes this possible at the architectural level is the interplay between the language model's in-context learning abilities and the visual grounding provided by cross-attention. The language model has already learned during text pretraining to identify and continue patterns in sequences. When you prepend visual examples to the context, the model maps each image's visual features (delivered via cross-attention) to the corresponding text labels. By the time it reaches the query image, the model has internally associated specific types of visual features with specific output patterns, effectively learning the task without updating any weights.

This is a remarkable demonstration of compositionality: the text-only language model contributes the pattern recognition and in-context learning capacity, the frozen vision encoder contributes perceptual representations, and the trainable Perceiver Resampler plus gating layers give the translation layer between them. No single component alone could reach this; the capability emerges from their combination.

One of the important empirical findings in the Flamingo paper is that performance consistently improves as the number of in-context examples increases from 0-shot to 1-shot to 4-shot to 8-shot, similar to the scaling behavior observed in GPT-3. This monotonic improvement suggests that the model uses the provided examples to adapt its behavior rather than simply relying on a fixed strategy. The rate of improvement also varies by task complexity: for simple recognition tasks, even 1-shot examples give most of the benefit, while for complex visual reasoning tasks, performance continues improving out to 8 or more shots.

The few-shot capability also generalizes across visual domains. A model given a few examples of medical imaging classifications can adapt to that domain without any medical imaging in its training data, provided the visual patterns are learnable from the frozen vision encoder's features. This domain generalization suggests that the cross-attention layers learn a general-purpose visual-to-linguistic translation mechanism rather than task-specific mappings.

Flamingo's training involves three distinct stages, each using different data sources but maintaining the same architectural principle: keep the vision encoder and language model frozen, train only the Perceiver Resampler and GATED XATTN-DENSE layers. This multi-stage curriculum progressively builds the model's capabilities, starting with general web-scale interleaved data, moving to higher-quality paired captions, and finally incorporating temporal video understanding.

The decision to use three separate stages rather than mixing all data jointly shows an important insight about how different data types contribute to different capabilities. Interleaved web data teaches contextual reasoning and few-shot pattern matching, since it contains natural demonstrations of how images relate to surrounding text in real documents. Paired caption data teaches fine-grained visual recognition, since high-quality captions tend to be more descriptive and accurate than casual textual references to images in web pages. Video data teaches temporal reasoning, since understanding dynamic scenes requires tracking how visual content changes across frames. Training on each type in sequence allows the model to build these skills progressively without the noisier data types overwhelming the signal from higher-quality sources.

The first stage trains on the Multimodal MassiveWeb dataset containing 183 million pages extracted from Common Crawl. Each training example consists of a sequence of up to 256 tokens of text interleaved with up to 5 images randomly sampled from the page. This stage teaches the model the basic skill of associating text with visual content in naturalistic contexts. The web data gives enormous diversity, covering everything from news articles and blog posts to instructional guides and forum discussions. This keeps the model learns reliable, generalizable associations between language and vision rather than domain-specific patterns.

The loss function is standard cross-entropy on the text tokens, measuring the negative log-likelihood of each token given the previous context and visual information:

where:

• $\mathcal{L}$: the total cross-entropy loss across the sequence
• $x_t$: the target text token at position $t$
• $x_{<t}$: all preceding text tokens in the sequence (the autoregressive context)
• $\text{Images}$: all images in the sequence, accessed through cross-attention
• $P(x_t \mid x_{<t}, \text{Images})$: the probability assigned to the correct token by the model, conditioned on both text history and visual features

This objective teaches the model to predict text tokens while grounding predictions in visual content, forcing the cross-attention mechanisms to learn real associations between linguistic concepts and visual features. The model must learn to attend to relevant images when generating descriptive text, ignore irrelevant images when they don't support the current context, and integrate information across multiple images when the text references comparisons or sequences.

A subtle but important aspect of this training objective is that only text tokens contribute to the loss. Image tokens never appear as prediction targets. This means the model is never explicitly trained to describe images; it is instead trained to predict what text should surround images in natural documents. The visual grounding capabilities emerge indirectly, as a consequence of the model learning to make accurate text predictions that are consistent with visual context. This indirect training signal turns out to be remarkably effective, creating visual understanding that generalizes well to tasks like visual question answering even though the model was never explicitly trained on question-answer pairs.

The second stage incorporates 1.8 billion image-text pairs from the ALIGN dataset. While interleaved data teaches contextual reasoning and few-shot learning, paired data gives high-quality single-image captions that improve fine-grained visual recognition. ALIGN consists of image-alt_text pairs filtered from web pages using contrastive learning to ensure high quality and diversity.

During this stage, each example contains exactly one image and its corresponding caption, letting the model to focus on detailed visual understanding without the complexity of cross-image reasoning. This stage particularly improves the model's ability to recognize specific objects and attributes alongside spatial relationships, as the captions in ALIGN tend to be more descriptive and accurate than the surrounding text in arbitrary web documents.

The transition from interleaved to paired training is a shift in the complexity of the signal. Interleaved data contains implicit associations between images and text (the connection between an image and the nearby text must be inferred from context), while paired data contains explicit one-to-one correspondences. By first learning implicit associations and then reinforcing them with explicit pairings, the curriculum mirrors effective educational approaches: start with detailed contextual learning, then consolidate with targeted practice.

The final stage adds 27 million video-text pairs from the VideoTextPairs dataset. Videos are processed as sequences of frames (sampled at 1 FPS), with each frame treated as a separate image. This extends Flamingo's capabilities to temporal reasoning and video understanding, though it remains primarily an image-text model with video capabilities emerging from frame-level processing.

By treating video frames as individual images processed through the same Perceiver Resampler, Flamingo learns to track objects and actions across time, understand temporal transitions, and generate descriptions of dynamic scenes. This capability, while emergent rather than explicitly architected for video, shows the flexibility of the Flamingo framework in handling sequential visual data.

The video training stage also reveals an interesting limitation of the Perceiver Resampler design. Because each frame is independently processed through the resampler, the compressed visual tokens for each frame are computed independently, without any cross-frame attention. This means the resampler cannot directly learn motion-sensitive features that would require comparing adjacent frames. The temporal reasoning must emerge entirely through the language model's cross-attention to the sequence of per-frame visual tokens, which works reasonably well for slow or predictable motion but may miss fine-grained temporal details.

Training uses the AdamW optimizer with decoupled weight decay, similar to what we discussed in Part X, Ch 9: AdamW. The learning rate warms up linearly and then decays with a cosine schedule. Different components use different learning rates: the Perceiver Resampler and GATED XATTN-DENSE layers use a higher learning rate ($10^{-4}$ to $10^{-3}$) than the gating parameters ($10^{-5}$). This keeps stable initial training while letting the gates to open gradually.

This learning rate differentiation is important: if the gating parameters $\alpha$ were updated too quickly, the gates might swing open rapidly, causing sudden distribution shifts in the frozen language model's activations and potentially destabilizing training. By keeping the gating learning rate low, the model ensures smooth, gradual adaptation of the residual connections.

Gradient clipping is applied throughout training, as we discussed in Part X, Ch 13: Gradient Clipping. Because the frozen language model and vision encoder contribute gradients to the trainable layers through the chain rule (even though their own parameters aren't updated), the gradients flowing through the GATED XATTN-DENSE layers can be large and unstable early in training. Clipping to a maximum gradient norm of 1.0 prevents these large gradients from causing sudden parameter updates that could disrupt the gating mechanism.

The training uses a batch size that balances the three data sources. Specifically, the Flamingo paper reports using a batch that combines M3W, ALIGN, and VTP examples in ratios that give each data source roughly equal importance in the gradient updates despite their very different sizes. This weighting prevents the much larger ALIGN dataset from dominating training while still benefiting from its higher data quality.

Let's implement the core components of the Flamingo architecture. We'll construct a simplified version that shows the Perceiver Resampler and Gated Cross-Attention mechanisms, then show how they integrate with a pretrained language model.

The Perceiver Resampler uses learned latent queries to compress vision features. We'll implement it with configurable depth (number of layers) and the ability to handle variable input lengths.

The resampler successfully compresses 196 vision tokens into 64 visual tokens, a compression ratio of roughly 3x. This compression is important for managing computational costs when processing multiple high-resolution images. Without this compression, processing a sequence of 10 images would require the language model to attend to 1,960 visual tokens, creating a significant computational bottleneck in the cross-attention layers. The 9.4x reduction in attention cost comes from the quadratic nature of attention: halving the number of tokens reduces attention cost by a factor of four, and reducing by 3.1x reduces it by more than nine.

Now let's implement the GATED XATTN component. This layer allows text tokens to attend to visual tokens while using the gating mechanism to control information flow.

Initially, the gate is effectively closed ($\tanh(0) = 0$), meaning the output equals the input. This preserves the language model's behavior at the start of training. As $\alpha$ increases (simulated here by setting it to 2.0), the gate opens to approximately 0.96, letting visual information to flow through. This smooth transition from identity to full transformation lets stable optimization, preventing the shock to the system that would occur if visual information were injected at full strength before the model learned how to interpret it.

Now we combine the cross-attention with a feed-forward network to create the full GATED XATTN-DENSE block used in Flamingo.

Let's see how these components fit together in a forward pass with interleaved text and images.

This shows how visual tokens are integrated into the text processing stream. The text maintains its autoregressive structure (enforced by the causal mask), while cross-attention allows every text position to attend to all visual tokens from preceding images. This architecture lets the model to reference visual information at any point during text generation, whether describing specific details, answering questions about image content, or comparing multiple images in sequence.

The key parameters for the Flamingo components are:

• dim: Model dimension (d_model) used throughout all components. Must match between vision resampler and language model layers to ensure compatibility of tensor operations and attention mechanisms.
• num_latents: Number of output visual tokens from the Perceiver Resampler (typically 64). Controls the compression ratio of vision features and the computational cost of cross-attention in the language model.
• num_layers: Number of self-attention layers within the Perceiver Resampler. More layers allow richer interactions among visual tokens but increase computation during the vision processing phase.
• num_heads: Number of attention heads in cross-attention and self-attention mechanisms. More heads let attending to different aspects of visual features simultaneously, improving the richness of visual representation.
• ff_mult: Feed-forward network expansion factor in GATED XATTN-DENSE layers. Larger values increase capacity but add parameters, affecting memory requirements and training time.
• alpha: Learnable gating parameter initialized to 0, controlling the flow of visual information through tanh gating. The final values of alpha after training indicate how strongly each layer conditions on visual information.

The gating mechanism proves needed for training stability. When we initialize $\alpha = 0$, the GATED XATTN-DENSE layers initially perform identity operations. This means that at the start of training, the Flamingo model behaves exactly like the frozen base language model, creating identical hidden states and output distributions. This identity initialization gives a important baseline: the model begins from a known good configuration where language generation is already fluent and coherent, then gradually learns to incorporate visual information without sacrificing this baseline capability.

As training progresses, the gates gradually open. This slow adaptation prevents the sudden distribution shifts that would occur if we directly inserted untrained cross-attention layers into a frozen, optimized network. The language model's representations have been trained to expect specific activation statistics; abruptly injecting noisy visual gradients would disrupt these carefully tuned dynamics. The smooth, continuous opening of the gates acts as a curriculum for the model, letting it to first learn simple visual associations before tackling complex multimodal reasoning.

The training loss curves reported in the Flamingo paper show no initial spike or instability, contrasting with alternative approaches that fine-tune the entire model or insert ungated adapters. This stability allows Flamingo to benefit from the full pretrained capabilities of both its vision encoder and language model while learning their interaction. The gating parameters typically converge to values between 0.5 and 0.9, showing that the model learns to heavily condition on visual information while maintaining the residual connection's stabilizing effect.

The differential convergence rates across layers tell an interesting story about how visual information is integrated through the network. Early layers tend to open their gates faster and to higher values. This makes intuitive sense: the early layers of the language model process lower-level token representations where directly incorporating visual grounding is straightforward (associating visual regions with noun tokens, for instance). Deeper layers deal with more abstract semantic representations where the relationship between visual features and text is more complex, requiring more training data to establish reliable cross-modal associations.

Flamingo was evaluated on a complete suite of vision-language benchmarks, establishing state-of-the-art performance at the time of publication across multiple tasks. Understanding these results helps clarify what the architecture reaches in practice and where its strengths and weaknesses lie.

On the VQAv2 benchmark, which tests visual question answering on a diverse set of natural images, the 80B Flamingo model achieved 82.0% accuracy in a 32-shot setting, surpassing the best fine-tuned models at the time despite not using any task-specific training data. This is a remarkable result: a model that has never been explicitly trained to answer questions about images outperforms models that have been fine-tuned for exactly this task. The performance advantage grows with the number of in-context examples. This shows that Flamingo uses the provided context rather than relying purely on patterns memorized during pretraining.

The few-shot VQA results also scale cleanly with model size. The 3B, 9B, and 80B Flamingo variants show consistent improvement as scale increases, suggesting that the visual reasoning capabilities benefit from larger language model capacity in ways consistent with the scaling laws we discussed in Part XXIII. This scaling behavior validates that the frozen language model's increased capacity contributes directly to multimodal reasoning, not just text generation quality.

On COCO Captions, Flamingo reaches strong CIDEr scores in few-shot settings even without access to ground-truth captions during evaluation. The captions generated by Flamingo tend to be more descriptive and contextually appropriate than simpler captioning models. This shows the language model's advanced linguistic generation capabilities. However, the model does not always match specialized fine-tuned captioning models on standard automatic metrics like BLEU and METEOR, partly because the language model's style of captioning (influenced by web text patterns) does not perfectly match the crowd-sourced caption style that these metrics reward.

One surprising strength of Flamingo is its performance on tasks requiring text reading from images, such as reading charts and signs as well as documents. Because the vision encoder captures fine-grained spatial features and the language model has extensive exposure to structured text formats, Flamingo can often correctly read and reason about text embedded in images using only a few in-context examples showing the task format. This capability, sometimes called visual text recognition, was not explicitly targeted during training but emerges from the combination of capable visual and linguistic components.

The consistent improvement from 0-shot to 4-shot shows real in-context learning across all four task categories. The gains are particularly large on TextVQA (which requires reading text in images), suggesting that in-context examples are especially helpful for tasks where the model needs to adapt its visual processing strategy. On VQAv2, even the 0-shot performance is competitive with some fine-tuned baselines, showing that the general visual-linguistic associations learned during multimodal pretraining transfer broadly.

The Flamingo paper introduced three model sizes, each using progressively larger frozen language models:

Flamingo-3B pairs an NFNet vision encoder with a 1.4B parameter language model. This variant is the most computationally accessible and shows that the architectural approach works even at smaller scales. The 3B model reaches competitive few-shot performance on most benchmarks, confirming that the Flamingo approach is not exclusively a large-model phenomenon.

Flamingo-9B uses the same vision encoder with a larger 7B parameter language model. The jump from 3B to 9B yields significant performance improvements across all benchmarks, particularly on tasks requiring complex visual reasoning. This improvement is consistent with the language model's increased capacity for multi-step reasoning and richer in-context pattern recognition.

Flamingo-80B is the full-scale model, pairing the NFNet encoder with an 80B parameter Chinchilla language model. This variant reaches the strongest performance across all evaluated tasks and most clearly shows the few-shot learning capabilities. The scaling from 9B to 80B yields smaller relative improvements than from 3B to 9B, consistent with the diminishing returns observed in language model scaling studies.

The parameter efficiency is clear when comparing Flamingo to end-to-end trained models of similar scale. Flamingo-80B, with roughly 10B trainable parameters (out of 80B total), matches or surpasses models with 80B fully trainable parameters on many benchmarks, suggesting that the pretrained representations in the frozen components are substantially more useful than additional randomly initialized parameters would be.

While Flamingo established a powerful paradigm for efficient multimodal learning, it carries several limitations that subsequent research has attempted to address.

Computational Complexity: Although the Perceiver Resampler reduces the number of visual tokens, Flamingo still suffers from quadratic complexity with respect to sequence length in the language model component, and the cross-attention adds an additional $O(T \cdot N \cdot D)$ term where $T$ is text length, $N$ is the number of visual tokens, and $D$ is the model dimension. For long videos with many frames, this becomes prohibitive. The attention mechanism must compute similarities between every text token and every visual token, creating a computational bottleneck as the number of images increases. Later models like LLaVA (which we introduced in Part L, Ch 5: LLaVA Architecture) use simpler projection layers to avoid this bottleneck, trading some expressiveness for computational efficiency.

Frozen Visual Representations: By freezing the NFNet vision encoder, Flamingo cannot adapt its visual representations to better suit the language model's needs. If the vision encoder produces features that aren't easily interpretable by the language model, the Perceiver Resampler must compensate for this mismatch within its limited capacity. This constraint means that Flamingo cannot learn new visual concepts not captured by the frozen encoder, nor can it optimize visual features for linguistic alignment. Future work explored unfreezing vision encoders or using vision transformers specifically pretrained for alignment with language, letting end-to-end optimization of the visual backbone.

The frozen encoder limitation becomes especially apparent for specialized visual domains. A model trained to recognize everyday objects may produce features that don't cleanly encode the distinctive characteristics of medical images, astronomical photographs, or microscopy slides. In these domains, Flamingo's visual representations may be systematically weaker than a model fine-tuned with domain-specific visual data, because the frozen encoder cannot reorganize its feature space in response to the downstream linguistic objectives.

Data Contamination: Training on web-scale interleaved data inevitably includes evaluation benchmarks in the training corpus. While Flamingo employs careful deduplication, the sheer scale of M3W makes perfect contamination control impossible. This complicates few-shot evaluation, as the model may have seen similar examples during training, potentially inflating performance metrics on standard benchmarks. Researchers must carefully design evaluation protocols to account for this contamination, often using temporal cutoff dates or aggressive filtering to ensure fair evaluation. The contamination concern is particularly acute for Flamingo given its few-shot evaluation methodology: if the model has memorized benchmark examples, even its 0-shot performance could reflect recall rather than generalization.

Context Window Limitations: Flamingo's ability to perform few-shot learning depends on fitting exemplars within the language model's context window (typically 2,048 tokens). For tasks requiring many examples or detailed visual reasoning chains, this limit constrains performance. Complex visual reasoning tasks might require dozens of examples to learn effectively, exceeding the capacity of the context window. This limitation has motivated research into more efficient few-shot learning mechanisms and the development of models with longer context windows capable of handling extensive visual demonstrations.

Hallucination and Grounding Failures: Like other vision-language models, Flamingo can generate plausible-sounding text descriptions that are factually incorrect about the visual content. Because the language model component is frozen and optimized for fluency, it can sometimes generate confident, grammatically correct statements that don't accurately describe the input image. The cross-attention mechanism gives visual grounding, but it does not guarantee factual accuracy, especially for tasks requiring fine-grained spatial reasoning or precise quantitative judgments about visual content.

Despite these limitations, Flamingo's impact on the field has been significant. It demonstrated that frozen foundation models can be composed through lightweight, trainable interfaces, a concept that extends beyond vision-language modeling to audio and video as well as robotic control. The "Flamingo approach" (freezing pretrained components and learning adaptive gating layers) influenced subsequent architectures like OpenFlamingo, IDEFICS, various video-language models.

OpenFlamingo, released by LAION-AI as an open-source implementation, made the architectural principles accessible to the broader research community and enabled a wave of follow-up work exploring variations on the gating mechanism, different vision encoders, and alternative training data mixtures. IDEFICS (Images and Documents to Extract Contextualized Knowledge for In-Context learning) replicated and extended Flamingo's capabilities using openly available data sources. This shows that the approach does not require the proprietary datasets used in the original work.

Perhaps most materially, Flamingo validated the feasibility of few-shot learning in the visual domain, proving that the in-context learning capabilities we observed in large language models could generalize to multimodal reasoning. This opened research directions into visual prompt engineering and multimodal chain-of-thought reasoning that continue to evolve today. By showing that pretrained models can be taught new sensory modalities without forgetting their existing knowledge, Flamingo established a template for the rapid expansion of foundation model capabilities across diverse domains.

The architectural principles also influenced the broader ecosystem of multimodal AI beyond direct successors. The idea that a frozen, capable language model can serve as the "general intelligence" component while lightweight trained adapters give modality-specific processing has been applied to audio (as in AudioPaLM), video (as in VideoCoca), and robotics (as in RT-2). In each case, the frozen language model contributes language understanding and reasoning, while small trainable bridges connect it to new sensory inputs or action spaces. This modular perspective on building capable multimodal systems, enabled by Flamingo's demonstration that it was feasible, has become one of the dominant paradigms in contemporary AI systems research.

Flamingo introduced a blueprint for adapting large pretrained language models to handle visual inputs without catastrophic forgetting or prohibitive training costs. The architecture centers on three key innovations:

• Perceiver Resampler: Compresses variable-length vision encoder outputs into fixed visual tokens using learned latent queries and cross-attention, solving the dimensionality mismatch between vision and language modalities. This component lets efficient processing of high-resolution images by reducing thousands of visual features to a manageable number of tokens, making downstream cross-attention computationally tractable.

• GATED XATTN-DENSE Layers: Insert trainable cross-attention and feed-forward blocks between frozen language model layers, using tanh-gating initialized to zero for stable training. These layers condition text generation on visual content while preserving the base model's linguistic capabilities, acting as carefully controlled information channels between modalities that gradually open during training to incorporate visual information.

• Interleaved Training: Training on natural web documents containing sequences of images and text lets few-shot visual learning. This allows the model to perform new vision-language tasks from just a few examples in context. This training paradigm mirrors how humans learn from multimedia content. This gives a naturalistic foundation for multimodal understanding that generalizes across tasks without task-specific fine-tuning.

By freezing the vision encoder and language model while training only the lightweight resampler and gating layers, Flamingo reaches strong data efficiency, training on far less multimodal data than end-to-end alternatives while matching or exceeding their performance on vision-language benchmarks. The parameter-efficient adaptation strategy has become a cornerstone of modern multimodal AI. This shows that the right architectural interface between pretrained components can enable capabilities that neither component possesses independently. This insight has influenced the design of nearly every major vision-language model that followed. This makes Flamingo one of the most consequential architectural contributions of the modern era of multimodal AI.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Flamingo's architecture, gating mechanisms, and training paradigm.