Image Understanding › Image Captioning

Image captioning asks a model to generate a natural language description of an image's content. Unlike VQA, there is no question to guide the output. The model must independently decide what to describe: the main objects, their attributes, their spatial arrangement, the actions occurring, and the overall scene context.

A good caption is both accurate and fluent. Accuracy means the described content appears in the image and the described relationships are correct. Fluency means the text reads naturally, uses appropriate vocabulary, and forms grammatically correct sentences. Balancing these two requirements has driven most of the architectural evolution in captioning systems.

There is also a third, subtler requirement: informativeness. A caption that says "An image shows objects" is accurate and grammatical but useless. A caption that says "Two gray cats with orange tabby markings lie on a pink couch, one stretched horizontally and one curled with its head near the armrest" is accurate and grammatical while conveying useful detail. Models must learn to select the most salient and specific details, which requires knowing what matters and what does not.

Early neural image captioning systems, such as "Show and Tell" (Vinyals et al., 2015), adapted the encoder-decoder architecture from machine translation. A CNN encodes the image into a fixed-size vector, which initializes an LSTM that generates the caption word by word. This architecture was elegant and surprisingly effective given its simplicity, because the key infrastructure was already in place: CNNs had learned rich visual representations through ImageNet classification, and LSTMs had demonstrated strong performance on language modeling tasks.

Building on the sequence-to-sequence ideas from chapters on recurrent neural networks and attention, the key insight was that caption generation is a conditional language modeling problem. Given the image $I$, the model learns to maximize the probability of the correct caption $w_1, w_2, \ldots, w_T$:

$$P(w_1, \ldots, w_T \mid I) = \prod_{t=1}^{T} P(w_t \mid w_1, \ldots, w_{t-1}, I)$$

where:

• $P(w_t \mid w_1, \ldots, w_{t-1}, I)$: probability of the $t$-th word given all previous words and the image
• The product structure reflects the autoregressive generation of text left-to-right
• The image $I$ conditions the entire generation, giving visual context at every step

Training maximizes the log probability of reference captions given their associated images, which is equivalent to minimizing the cross-entropy loss summed over all positions in all training captions. The CNN visual encoder is often pretrained on ImageNet and fine-tuned during captioning training, letting its representations to shift toward the kinds of visual features most useful for generating descriptive language.

The CNN-to-LSTM pipeline worked because CNNs had already learned rich visual representations, and those representations, even though trained for classification, captured object and scene information needed for captioning. But the fixed-vector bottleneck became a clear limitation as caption complexity increased.

The fixed-vector bottleneck limited captioning quality. Using a single vector to represent an entire image means that when describing "the dog chasing the frisbee," the same vector must somehow encode both the dog, the frisbee, and their spatial relationship. Spatial information is inevitably compressed.

The "Show, Attend and Tell" model (Xu et al., 2015) introduced attention to captioning. Instead of encoding the image to a single vector, the CNN produces a spatial grid of features $\{a_1, a_2, \ldots, a_L\}$ where each $a_i$ represents a patch of the image. At each generation step $t$, the decoder computes attention weights over these spatial features, selecting which parts of the image to focus on when generating the next word.

The attention mechanism produces a weighted combination of spatial features:

$$c_t = \sum_{i=1}^{L} \beta_{t,i} \, a_i$$

where $\beta_{t,i}$ is the attention weight for spatial location $i$ at time step $t$, computed by a small neural network that takes the previous decoder hidden state and each spatial feature as inputs. The context vector $c_t$ is then combined with the decoder's hidden state to predict the next word probability distribution.

When generating "dog," attention concentrates on the dog's region. When generating "frisbee," attention shifts to the frisbee. When generating "chasing," attention might spread across the spatial gap between them. The caption reads as if the model is looking at different parts of the image as it writes, because it literally is.

Two forms of attention were proposed in Show, Attend and Tell: soft attention, which produces differentiable smooth weights over all spatial locations through the standard softmax operation, and hard attention, which stochastically selects a single location at each step. Soft attention is more commonly used because it can be trained with standard backpropagation, while hard attention requires policy gradient methods.

The bottom-up attention approach (Anderson et al., 2018) replaced spatial grid features with object-centric region features extracted by an object detector. This was a substantial improvement because natural language descriptions are inherently object-centric: we say "a woman eating a sandwich" rather than describing colors at spatial coordinates.

Object detection gives the captioning model pre-segmented, semantically meaningful regions to attend to. When describing a complex scene with multiple people and objects, region features make it much easier to produce accurate descriptions because each feature vector already corresponds to a recognized entity. The detector provides visual features and semantic priors: a vector labeled as "sandwich region" is more likely to elicit food-related vocabulary than a generic spatial patch.

This approach also revealed that the detector's training data becomes part of the captioning system's vocabulary. A detector trained on COCO categories can identify 80 object classes. A captioning model using those features can reliably describe those 80 categories but may struggle with objects outside the detector's vocabulary. This coupling between detector and captioner is both a strength (better object recognition) and a constraint (limited generalization).

Modern captioning systems are fully transformer-based. The image encoder is a Vision Transformer (ViT) or CLIP visual encoder, as we covered in the Vision Transformer chapter. The decoder is a standard transformer decoder that performs cross-attention over the visual token sequence.

The training objective remains the same autoregressive language modeling loss, but the architecture allows much richer cross-modal interaction. Every layer of the decoder can attend to every visual token, with different layers potentially specializing in different aspects: low-level appearance at early layers, object identity at intermediate layers, spatial relationships and scene-level context at later layers. This depth of interaction was not possible with the shallow single-step fusion of early CNN-LSTM systems.

BLIP-2 (Bootstrapping Language-Image Pre-training 2) represents a particularly clean example of modern captioning architecture. It uses a frozen image encoder (ViT-L from CLIP or EVA-CLIP) and a frozen language model (OPT or FlanT5), connected by a learned Q-Former module that extracts a small set of query vectors summarizing the visual content most relevant to language tasks. By keeping both the visual encoder and language model frozen, BLIP-2 can use powerful pretrained components without catastrophic forgetting, and the Q-Former learns to bridge the representation gap between them.

Automatic evaluation of captions is difficult. A caption can be perfectly accurate in different words than the reference. "A brown dog runs across a green field" and "A canine sprints through grass" describe the same image but share almost no vocabulary.

The standard evaluation metrics each capture different aspects:

• BLEU (Bilingual Evaluation Understudy): Measures n-gram overlap with reference captions. Fast and interpretable but penalizes valid paraphrases heavily.
• METEOR: Adds stemming and synonym matching alongside paraphrase matching on top of n-gram comparison. Better at catching semantic equivalence.
• CIDEr (Consensus-based Image Description Evaluation): Weights n-gram matches by their TF-IDF score across the full caption dataset. Penalizes generic phrases like "a man is" and rewards specific, information-rich content.
• SPICE (Semantic Propositional Image Caption Evaluation): Parses both generated and reference captions into scene graphs, then compares the graph structure. More reliable to paraphrase but computationally expensive.

No single metric captures all aspects of caption quality. In practice, CIDEr correlates best with human judgment on standard benchmarks, so it has become the primary optimization target in most captioning research. A concerning side effect of this is that models optimized directly for CIDEr, using reinforcement learning with CIDEr as the reward, produce captions that score well on the metric but strike human readers as formulaic or oddly specific in ways that match training data patterns rather than actual image content.

More recent work on caption evaluation uses large language models as judges. A judge model receives the generated caption, the reference captions, and sometimes the image itself, then rates its accuracy and fluency alongside informativeness on numerical scales. This approach correlates more strongly with human judgments than any single automatic metric, though it introduces its own biases depending on which language model is judge.