Flamingo Architecture › The Challenge of Adapting Frozen Models

Before diving into Flamingo's mechanics, let's understand why simply concatenating vision and language models proves insufficient. When we discussed cross-attention in Part XVI, Ch 4: Cross-Attention, we explored how transformers can attend across different modalities. However, standard cross-attention assumes both modalities use compatible representations. Vision encoders typically output feature maps with thousands of tokens (for example, 7x7 or 14x14 grids with high-dimensional features), while language models expect compact token sequences. Directly feeding thousands of visual tokens into every layer of a language model creates computational bottlenecks and disrupts the model's linguistic priors.

The computational challenge is severe: if a vision encoder produces 1,000 tokens per image and we process sequences with multiple images, the attention complexity grows quadratically with the total sequence length. More insidiously, the statistical properties of vision features differ fundamentally from text embeddings. Vision encoders capture spatial hierarchies of edges and textures alongside objects, while language models process sequential symbolic representations. Without careful alignment mechanisms, forcing these incompatible representations to interact can corrupt the carefully tuned feature spaces of either modality.

Consider what happens at the activation level. A pretrained language model has spent hundreds of billions of training tokens learning to produce specific ranges of intermediate activations. The mean and variance of hidden states at each layer have been implicitly optimized so that layer normalizations, feed-forward networks, and attention mechanisms work harmoniously. When you inject visual features with completely different statistical properties into this carefully calibrated system, the layer normalizations begin operating in unexpected regimes, the feed-forward networks receive inputs outside their effective domain, and the attention patterns learned during text pretraining break down. The model does not just fail to understand images; it forgets how to process language.

Training a vision-language model from scratch also requires enormous datasets and computational resources. As we saw in Part XXIII: Scaling Laws, model performance depends heavily on scale, yet retraining billion-parameter models for every new modality combination is economically impractical. The financial and environmental costs of pretraining large transformers from scratch can reach millions of dollars per run. Flamingo asks: can we use existing, capable vision encoders and language models while teaching them to communicate effectively?

Several obvious approaches fail in instructive ways. The simplest idea is to just prepend image tokens to the text sequence and let the language model's self-attention handle everything. This approach has two fatal flaws. First, it requires the language model's input embedding layer to accept a new token type, which means either modifying the pretrained embedding weights (risking forgetting) or projecting vision features into the text embedding space, which gives no mechanism for the model to distinguish visual from textual tokens. Second, the number of image tokens can be enormous (hundreds to thousands for high-resolution images), and forcing the language model to attend to all of them via self-attention at every layer sharply increases computational cost without any architectural safeguard against catastrophic forgetting.

Another approach is to fine-tune the entire model end-to-end on multimodal data, in effect treating vision-language adaptation as standard transfer learning. This approach works when you have enough multimodal data and computational budget, but it is expensive and wasteful: you're spending enormous resources to adjust parameters that were already optimized and shouldn't need to change. More critically, fine-tuning at scale tends to overwrite the fine-grained linguistic capabilities built up during text pretraining, particularly for low-frequency vocabulary, complex reasoning chains, and stylistic register that are underrepresented in typical image-caption datasets.

A third approach uses adapters placed inside the language model, similar to what we explored in Part XXXV: Parameter-Efficient Fine-Tuning. These approaches partially address the parameter efficiency problem, but standard adapters don't naturally handle the sequential, multi-image structure of visual inputs, and they don't address the variable-length output from vision encoders. Flamingo's contribution is to combine the insights from adapter research with a purpose-built compression mechanism and a carefully designed cross-modal attention architecture.

The answer lies in adapter-based architectures that introduce trainable bottlenecks between frozen components. By treating the vision and language models as fixed feature extractors and learning only the interface between them, we minimize the risk of catastrophic forgetting while maximizing data efficiency.