Flamingo Architecture › Summary

Flamingo introduced a blueprint for adapting large pretrained language models to handle visual inputs without catastrophic forgetting or prohibitive training costs. The architecture centers on three key innovations:

• Perceiver Resampler: Compresses variable-length vision encoder outputs into fixed visual tokens using learned latent queries and cross-attention, solving the dimensionality mismatch between vision and language modalities. This component lets efficient processing of high-resolution images by reducing thousands of visual features to a manageable number of tokens, making downstream cross-attention computationally tractable.

• GATED XATTN-DENSE Layers: Insert trainable cross-attention and feed-forward blocks between frozen language model layers, using tanh-gating initialized to zero for stable training. These layers condition text generation on visual content while preserving the base model's linguistic capabilities, acting as carefully controlled information channels between modalities that gradually open during training to incorporate visual information.

• Interleaved Training: Training on natural web documents containing sequences of images and text lets few-shot visual learning. This allows the model to perform new vision-language tasks from just a few examples in context. This training paradigm mirrors how humans learn from multimedia content. This gives a naturalistic foundation for multimodal understanding that generalizes across tasks without task-specific fine-tuning.

By freezing the vision encoder and language model while training only the lightweight resampler and gating layers, Flamingo reaches strong data efficiency, training on far less multimodal data than end-to-end alternatives while matching or exceeding their performance on vision-language benchmarks. The parameter-efficient adaptation strategy has become a cornerstone of modern multimodal AI. This shows that the right architectural interface between pretrained components can enable capabilities that neither component possesses independently. This insight has influenced the design of nearly every major vision-language model that followed. This makes Flamingo one of the most consequential architectural contributions of the modern era of multimodal AI.