Flamingo Architecture › Flamingo Architecture

Flamingo shifts how we approach multimodal AI. This shows that large language models can acquire visual understanding without catastrophic forgetting of their linguistic capabilities. Rather than training a vision-language model from scratch, Flamingo inserts lightweight trainable components between a frozen vision encoder and a frozen language model. This makes possible the resulting system to process arbitrarily interleaved sequences of text and images while maintaining the ability to perform few-shot learning across vision-language tasks.

This approach differs from earlier methods that typically required joint pretraining of vision and language components from random initialization. By preserving the pretrained weights of both modalities, Flamingo uses the large knowledge already encoded in state-of-the-art vision encoders and language models, avoiding the immense data and computational requirements of training billion-parameter models from scratch. The resulting system exhibits versatility, capable of engaging in visual question answering, image captioning, and multimodal dialogue while retaining the fluency and reasoning abilities of its base language model.

The architecture addresses a core challenge in multimodal AI: how to add new sensory modalities to a capable language model without destroying its existing knowledge or requiring prohibitively expensive end-to-end training. This tension arises because language models develop highly specific internal representations through extensive pretraining on text. When new input modalities are introduced, the distribution of activations can shift sharply, causing the model to forget previously learned linguistic patterns, a phenomenon known as catastrophic forgetting. Flamingo's solution relies on two key innovations: the Perceiver Resampler, which compresses high-dimensional visual features into a fixed number of tokens, and Gated Cross-Attention Dense (GATED XATTN-DENSE) layers, which condition the language model on visual information through carefully gated cross-attention mechanisms. These components are trained while the underlying vision and language models remain frozen, creating an efficient adaptation paradigm that has influenced subsequent vision-language models like OpenFlamingo and IDEFICS.