Flamingo Architecture › Few-Shot Visual Learning
Language AI HandbookAll chapters Flamingo Architecture › Few-Shot Visual LearningOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
One of Flamingo's most remarkable capabilities is few-shot in-context learning for vision-language tasks, analogous to the text-based in-context learning we explored in Part XXVIII, Ch 4: In-Context Learning. Just as GPT-3 can learn new text tasks from a few examples in the prompt, Flamingo can adapt to new vision tasks by observing a few image-text pairs, without any gradient updates or parameter modifications.
This capability emerges from training on interleaved image-text data from web pages. Unlike paired datasets where one image corresponds to one caption, web documents naturally contain sequences of images interspersed with explanatory text. This natural formatting mirrors the few-shot prompting paradigm, where examples are provided in context followed by a query. Flamingo is trained on the Multimodal MassiveWeb (M3W) dataset, which extracts such interleaved sequences from HTML documents, preserving the natural flow of visual and textual information as it appears in real-world documents.
A training paradigm where models learn from sequences containing multiple images and text passages in natural order, as they appear in documents, rather than isolated image-caption pairs. This teaches the model to use context from multiple images when generating text.
During training, the model sees sequences like:
[Image 1] The cat sat on the mat. [Image 2] Meanwhile, the dog slept on the couch. Both animals look comfortable.
The training objective remains standard next-token prediction on the text, but the model must attend to the relevant images through cross-attention to predict tokens accurately. For instance, to predict the word "cat," the model must associate the linguistic token with the visual features of the first image. To predict "Both animals," the model must integrate information from both images, learning cross-image referencing and coreference resolution in the visual domain. This teaches Flamingo to ground linguistic concepts in visual features and to maintain coherence across multiple images.
The Mechanics of Few-Shot Visual Inference
For few-shot evaluation, we give examples in the same interleaved format:
[Image: golden retriever] dog [Image: tabby cat] cat [Image: unknown breed] ?
The model learns to recognize the pattern (image to label) and applies it to the query image, effectively performing few-shot classification without any gradient updates or task-specific fine-tuning. This capability extends beyond simple classification to complex visual reasoning tasks, including visual question answering, optical character recognition, and commonsense reasoning about visual scenes, all from just a handful of examples provided in the context window.
What makes this possible at the architectural level is the interplay between the language model's in-context learning abilities and the visual grounding provided by cross-attention. The language model has already learned during text pretraining to identify and continue patterns in sequences. When you prepend visual examples to the context, the model maps each image's visual features (delivered via cross-attention) to the corresponding text labels. By the time it reaches the query image, the model has internally associated specific types of visual features with specific output patterns, effectively learning the task without updating any weights.
This is a remarkable demonstration of compositionality: the text-only language model contributes the pattern recognition and in-context learning capacity, the frozen vision encoder contributes perceptual representations, and the trainable Perceiver Resampler plus gating layers give the translation layer between them. No single component alone could reach this; the capability emerges from their combination.
Few-Shot Performance Scaling
One of the important empirical findings in the Flamingo paper is that performance consistently improves as the number of in-context examples increases from 0-shot to 1-shot to 4-shot to 8-shot, similar to the scaling behavior observed in GPT-3. This monotonic improvement suggests that the model uses the provided examples to adapt its behavior rather than simply relying on a fixed strategy. The rate of improvement also varies by task complexity: for simple recognition tasks, even 1-shot examples give most of the benefit, while for complex visual reasoning tasks, performance continues improving out to 8 or more shots.
The few-shot capability also generalizes across visual domains. A model given a few examples of medical imaging classifications can adapt to that domain without any medical imaging in its training data, provided the visual patterns are learnable from the frozen vision encoder's features. This domain generalization suggests that the cross-attention layers learn a general-purpose visual-to-linguistic translation mechanism rather than task-specific mappings.