Flamingo Architecture › Training Dynamics and Stability

The gating mechanism proves needed for training stability. When we initialize $\alpha = 0$, the GATED XATTN-DENSE layers initially perform identity operations. This means that at the start of training, the Flamingo model behaves exactly like the frozen base language model, creating identical hidden states and output distributions. This identity initialization gives a important baseline: the model begins from a known good configuration where language generation is already fluent and coherent, then gradually learns to incorporate visual information without sacrificing this baseline capability.

As training progresses, the gates gradually open. This slow adaptation prevents the sudden distribution shifts that would occur if we directly inserted untrained cross-attention layers into a frozen, optimized network. The language model's representations have been trained to expect specific activation statistics; abruptly injecting noisy visual gradients would disrupt these carefully tuned dynamics. The smooth, continuous opening of the gates acts as a curriculum for the model, letting it to first learn simple visual associations before tackling complex multimodal reasoning.

The training loss curves reported in the Flamingo paper show no initial spike or instability, contrasting with alternative approaches that fine-tune the entire model or insert ungated adapters. This stability allows Flamingo to benefit from the full pretrained capabilities of both its vision encoder and language model while learning their interaction. The gating parameters typically converge to values between 0.5 and 0.9, showing that the model learns to heavily condition on visual information while maintaining the residual connection's stabilizing effect.

The differential convergence rates across layers tell an interesting story about how visual information is integrated through the network. Early layers tend to open their gates faster and to higher values. This makes intuitive sense: the early layers of the language model process lower-level token representations where directly incorporating visual grounding is straightforward (associating visual regions with noun tokens, for instance). Deeper layers deal with more abstract semantic representations where the relationship between visual features and text is more complex, requiring more training data to establish reliable cross-modal associations.