Flamingo Architecture › Key Parameters
Michael BrenndoerferFebruary 12, 20261 min read
Language AI HandbookAll chapters Part 50 of 67Multimodal Models
Flamingo Architecture › Key ParametersOne section · Sections in this chapter7 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for the Flamingo components are:
- dim: Model dimension (d_model) used throughout all components. Must match between vision resampler and language model layers to ensure compatibility of tensor operations and attention mechanisms.
- num_latents: Number of output visual tokens from the Perceiver Resampler (typically 64). Controls the compression ratio of vision features and the computational cost of cross-attention in the language model.
- num_layers: Number of self-attention layers within the Perceiver Resampler. More layers allow richer interactions among visual tokens but increase computation during the vision processing phase.
- num_heads: Number of attention heads in cross-attention and self-attention mechanisms. More heads let attending to different aspects of visual features simultaneously, improving the richness of visual representation.
- ff_mult: Feed-forward network expansion factor in GATED XATTN-DENSE layers. Larger values increase capacity but add parameters, affecting memory requirements and training time.
- alpha: Learnable gating parameter initialized to 0, controlling the flow of visual information through tanh gating. The final values of alpha after training indicate how strongly each layer conditions on visual information.