Image Understanding: VQA, Captioning and Visual Grounding

What does it mean for a machine to truly understand an image? It means more than labeling it as "cat" or "dog": the system should answer questions about it, describe what is happening in rich natural language, locate specific objects in response to a query, and reason about the relationships between everything visible in the scene. This is the domain of image understanding, and it sits at the heart of modern multimodal AI.

The challenge is not primarily computational. High-resolution images can be processed efficiently with modern hardware, and deep neural networks can extract rich representations from pixel data. The harder challenge is semantic: connecting the model's internal image representation to concepts and relationships as well as reasoning patterns expressed in natural language. A person looking at a photo of a kitchen can immediately answer "Is there anything you could use to boil water?" because they recognize a kettle, know what kettles do, and can apply that functional knowledge to the question. Achieving the same capability in a machine requires connecting visual and linguistic information with commonsense reasoning.

As we explored in the Multimodal Foundations chapter, vision-language models learn to align visual and textual representations through large-scale pretraining. Image understanding tasks put those aligned representations to work. They test whether a model has learned to reason across modalities or simply memorized surface-level correlations. The four core tasks in this chapter, visual question answering, image captioning, visual grounding, and scene understanding, each probe a different dimension of this capability.

Visual question answering asks whether a model can correctly respond to targeted queries about specific image content. Image captioning asks whether the model can independently choose what to describe and express it in fluent natural language. Visual grounding asks whether the model can map language phrases to precise spatial locations. Scene understanding asks whether the model can build a complete, structured representation of the entire visual world in a single image.

This chapter covers the full spectrum of image understanding: the task definitions, the models designed to tackle them, the datasets that measure progress, and the architectural decisions that make these systems work. By the end, you will understand how the same underlying vision-language architecture can be adapted for asking questions about images, generating descriptions, pointing to specific objects, and parsing the compositional structure of a scene.

Visual question answering, commonly called VQA, is perhaps the most direct test of image understanding. Given an image and a natural language question about it, a model must produce a correct answer. The question might ask about color ("What color is the umbrella?"), count ("How many dogs are in the photo?"), spatial relationship ("Is the cup to the left or right of the plate?"), activity ("What sport is the person playing?"), or abstract reasoning ("Would it be safe to walk barefoot here?").

VQA requires integrating two fundamentally different information streams. The visual stream provides pixel-level detail about shapes and textures alongside positions and appearances. The textual stream provides the question: which aspect of the image matters, what concept to look for, what relationship to evaluate. Getting the answer right requires the model to identify the visually relevant region, apply the conceptual query from the text, and produce an output in the appropriate format.

What makes VQA especially demanding is that the question determines which parts of the image matter. "What is the person wearing?" requires focusing on the person's clothing. "How many windows are visible?" requires a systematic scan of the whole scene. "Is the light on or off?" requires locating a specific small object and reading its state. The model cannot know in advance what kind of attention the scene requires because that depends entirely on the question, which arrives alongside the image.

The original VQA dataset, introduced by Antol et al. in 2015, established the benchmark format that the field still uses today. Early models used CNN-extracted visual features combined with RNN-encoded question features, merged through element-wise multiplication or concatenation, and classified into one of the top-K answer classes. This framing treated VQA as a classification task: given a fixed vocabulary of answers, predict which one is correct.

The classification approach was natural for the era. CNNs trained on ImageNet had proven highly effective at encoding image content into fixed-size vectors. RNNs, particularly LSTMs, had become the standard tool for encoding variable-length question sequences into fixed-size context vectors. Combining these two vectors through a learned fusion function and predicting a class label was an elegant and direct application of existing technology to a new task.

The classification approach has a basic limitation, however: it cannot produce answers outside the vocabulary. Free-form VQA requires generative models. But classification VQA remains useful for benchmarking because it produces clean accuracy numbers, and the top-K answer vocabulary (typically around 3000 answers) covers the large majority of question types in standard datasets.

The core challenge in VQA is not extracting good visual features or good text features in isolation. Both of those problems are largely solved by the time you reach a VQA system. The challenge is fusing them correctly so that the answer reflects both the image content and the question semantics jointly.

Consider the question "What is the person wearing?" asked over an image showing a woman in a red dress with a blue hat. To answer correctly, the model must:

1. Locate the person in the image (visual localization)
2. Identify the person's clothing (visual recognition of a category)
3. Understand that "wearing" refers to clothing on the body (semantic disambiguation)
4. Produce "a red dress and a blue hat" (multi-attribute generation)

Each step depends on the previous, and each step requires both modalities. A purely visual model cannot know which person to focus on if multiple people are present. A purely textual model knows nothing about what the woman is wearing. The fusion must happen in a way that is sensitive to the interaction between them.

Early fusion approaches were relatively shallow. Element-wise multiplication of a visual feature vector and a question feature vector creates a combined representation where each dimension of the output reflects both visual and linguistic information. But this global fusion is blind to spatial structure: the entire image is collapsed into a single vector before any question-specific reasoning happens. The question "Is the red ball to the left or right of the blue ball?" requires comparing two object locations, but a global fusion vector represents the whole image rather than individual objects or their spatial relationships.

The breakthrough that dramatically improved VQA accuracy was applying cross-modal attention. Rather than fusing global image and text features once, attention allows the model to dynamically focus on the image regions most relevant to the question.

In the bottom-up and top-down attention model (Anderson et al., 2018), a bottom-up pathway uses an object detector (Faster R-CNN) to extract features for salient image regions: bounding boxes around objects and people as well as other visual entities. A top-down pathway processes the question and produces an attention query that selects which regions to attend to when creating the answer. This approach lets "What color is the apple?" focus attention on apple regions while "How many people are standing?" distributes attention across all person regions.

The attention query from the question changes what the visual representation emphasizes. For a question about a specific object, attention concentrates on that object. For a question about a scene-level property, attention distributes more broadly. This dynamic allocation of visual attention, guided by language, is exactly the kind of cross-modal reasoning VQA requires.

The mathematical structure mirrors the attention mechanism from transformers. Given region features $\{v_1, v_2, \ldots, v_k\}$ and question encoding $q$, attention weights are:

where:

• $\alpha_i$: the attention weight for region $i$, showing how much this region contributes to the answer
• $f(v_i, q)$: a compatibility function between region feature $v_i$ and question encoding $q$, often implemented as a dot product or a small neural network
• The softmax normalization ensures all weights are positive and sum to one across all $k$ regions

The attended visual context is the weighted sum $\hat{v} = \sum_i \alpha_i v_i$, a soft aggregation of all region features weighted by their relevance to the question. This attended vector, combined with the question encoding $q$, forms the joint representation used to predict the answer. The model learns what kind of attention to pay to each type of question entirely through gradient descent on the VQA training objective.

Simple element-wise multiplication captures first-order interactions between visual and language features, but many visual questions require higher-order reasoning. "Is the person wearing the same color as the car?" requires comparing the person's clothing color to the car's color, an operation that cannot be reduced to a single attended region.

Bilinear fusion methods like MCB (Multimodal Compact Bilinear pooling) and MLB (Multi-modal Low-rank Bilinear) compute richer interactions between visual and question vectors by approximating the full outer product of the two feature vectors. The full outer product of a visual vector $v \in \mathbb{R}^{d_v}$ and a question vector $q \in \mathbb{R}^{d_q}$ has $d_v \times d_q$ elements, which is far too large to be practical. Compact bilinear methods project both vectors into a shared high-dimensional space using random features, then compute element-wise products in that space. This approximates the full bilinear interaction at a fraction of the computational cost.

These richer fusion methods consistently outperformed simple concatenation or element-wise multiplication on VQA benchmarks, confirming that the nature of the multimodal interaction matters, not just which features are combined.

An important discovery in VQA research was that models could achieve surprisingly high accuracy by exploiting statistical regularities in the dataset, without reasoning about images at all. Questions starting with "Do you see a..." are answered "yes" much more often than "no" in the VQA v1 dataset. Questions about color are answered "white" more often than any other color. Counting questions rarely have answers above 5. A model that learned these priors could score reasonably well without looking at a single pixel.

This led to the creation of VQA v2 (Goyal et al., 2017), which pairs each question with two complementary images where the same question has different correct answers. "Is there a dog on the couch?" is paired with an image where there is a dog on the couch and an image where there is not. This pairing forces the model to distinguish between image contents rather than exploiting question-level statistics.

Even with balanced datasets, the language bias problem persists in subtler forms. The VQA-CP (changing priors) benchmark reorganizes VQA v2 so that the most common answer for each question type changes between training and test splits. A model that learned to answer color questions with "white" during training will fail on VQA-CP test questions where the correct color answers are primarily other colors. Many models that achieve strong performance on VQA v2 drop by 20-30 percentage points on VQA-CP, revealing how much of their "performance" depended on language priors rather than image reasoning.

Current state-of-the-art VQA systems are not specialized VQA architectures at all. They are large vision-language models, such as BLIP-2, LLaVA, or GPT-4V, that handle VQA as one of many tasks through instruction following. The model sees the image and the question as a combined input and generates the answer autoregressively.

This shift from classification to generation changes the training paradigm fundamentally. Instead of fine-tuning a VQA classification head, modern models are either pretrained on VQA-style data among many other tasks, or prompted at inference time with instructions like "Look at this image and answer the following question: {question}."

The advantage is generalization: a model trained this way can answer questions whose answers lie outside any predefined vocabulary, handle follow-up questions in a conversation, explain its reasoning, and express uncertainty. A VQA model based on a language model can also use the world knowledge encoded in its language pretraining to answer questions that require reasoning beyond the image itself. "What country was this flag designed in?" requires recognizing the flag visually and then retrieving the associated country from language model knowledge.

The disadvantage is evaluation complexity: comparing generated text to ground truth answers requires more sophisticated metrics than simple accuracy. "Two" and "2" should both be correct answers for a counting question. String matching fails on these cases; modern evaluation often uses language model judges to determine semantic equivalence.

Standard VQA datasets use a specific accuracy metric that accounts for human disagreement. Each question has 10 human-provided answers. The model's answer receives a score:

This formula gives full credit if at least 3 of 10 annotators agree with the model, partial credit for 1-2 agreements, and zero credit for an answer nobody gave. This handles questions where multiple answers are valid: "What sport is this?" might receive "tennis" from 7 annotators and "playing tennis" from 3, both of which should be considered correct. The minimum-with-3 normalization means the score caps at 1 even if all 10 annotators agree, since the point is sufficiency rather than popularity.

This graded scoring matters practically. A model that always answers "cat" to any question would score 0 on questions where no annotator said "cat" but might score $\frac{1}{3} \approx 0.33$ on questions where one annotator happened to say "cat." Average VQA accuracy on the validation set reflects the weighted sum of these per-question scores across the entire dataset.

Image captioning asks a model to generate a natural language description of an image's content. Unlike VQA, there is no question to guide the output. The model must independently decide what to describe: the main objects, their attributes, their spatial arrangement, the actions occurring, and the overall scene context.

A good caption is both accurate and fluent. Accuracy means the described content appears in the image and the described relationships are correct. Fluency means the text reads naturally, uses appropriate vocabulary, and forms grammatically correct sentences. Balancing these two requirements has driven most of the architectural evolution in captioning systems.

There is also a third, subtler requirement: informativeness. A caption that says "An image shows objects" is accurate and grammatical but useless. A caption that says "Two gray cats with orange tabby markings lie on a pink couch, one stretched horizontally and one curled with its head near the armrest" is accurate and grammatical while conveying useful detail. Models must learn to select the most salient and specific details, which requires knowing what matters and what does not.

Early neural image captioning systems, such as "Show and Tell" (Vinyals et al., 2015), adapted the encoder-decoder architecture from machine translation. A CNN encodes the image into a fixed-size vector, which initializes an LSTM that generates the caption word by word. This architecture was elegant and surprisingly effective given its simplicity, because the key infrastructure was already in place: CNNs had learned rich visual representations through ImageNet classification, and LSTMs had demonstrated strong performance on language modeling tasks.

Building on the sequence-to-sequence ideas from chapters on recurrent neural networks and attention, the key insight was that caption generation is a conditional language modeling problem. Given the image $I$, the model learns to maximize the probability of the correct caption $w_1, w_2, \ldots, w_T$:

where:

• $P(w_t \mid w_1, \ldots, w_{t-1}, I)$: probability of the $t$-th word given all previous words and the image
• The product structure reflects the autoregressive generation of text left-to-right
• The image $I$ conditions the entire generation, giving visual context at every step

Training maximizes the log probability of reference captions given their associated images, which is equivalent to minimizing the cross-entropy loss summed over all positions in all training captions. The CNN visual encoder is often pretrained on ImageNet and fine-tuned during captioning training, letting its representations to shift toward the kinds of visual features most useful for generating descriptive language.

The CNN-to-LSTM pipeline worked because CNNs had already learned rich visual representations, and those representations, even though trained for classification, captured object and scene information needed for captioning. But the fixed-vector bottleneck became a clear limitation as caption complexity increased.

The fixed-vector bottleneck limited captioning quality. Using a single vector to represent an entire image means that when describing "the dog chasing the frisbee," the same vector must somehow encode both the dog, the frisbee, and their spatial relationship. Spatial information is inevitably compressed.

The "Show, Attend and Tell" model (Xu et al., 2015) introduced attention to captioning. Instead of encoding the image to a single vector, the CNN produces a spatial grid of features $\{a_1, a_2, \ldots, a_L\}$ where each $a_i$ represents a patch of the image. At each generation step $t$, the decoder computes attention weights over these spatial features, selecting which parts of the image to focus on when generating the next word.

The attention mechanism produces a weighted combination of spatial features:

where $\beta_{t,i}$ is the attention weight for spatial location $i$ at time step $t$, computed by a small neural network that takes the previous decoder hidden state and each spatial feature as inputs. The context vector $c_t$ is then combined with the decoder's hidden state to predict the next word probability distribution.

When generating "dog," attention concentrates on the dog's region. When generating "frisbee," attention shifts to the frisbee. When generating "chasing," attention might spread across the spatial gap between them. The caption reads as if the model is looking at different parts of the image as it writes, because it literally is.

Two forms of attention were proposed in Show, Attend and Tell: soft attention, which produces differentiable smooth weights over all spatial locations through the standard softmax operation, and hard attention, which stochastically selects a single location at each step. Soft attention is more commonly used because it can be trained with standard backpropagation, while hard attention requires policy gradient methods.

The bottom-up attention approach (Anderson et al., 2018) replaced spatial grid features with object-centric region features extracted by an object detector. This was a substantial improvement because natural language descriptions are inherently object-centric: we say "a woman eating a sandwich" rather than describing colors at spatial coordinates.

Object detection gives the captioning model pre-segmented, semantically meaningful regions to attend to. When describing a complex scene with multiple people and objects, region features make it much easier to produce accurate descriptions because each feature vector already corresponds to a recognized entity. The detector provides visual features and semantic priors: a vector labeled as "sandwich region" is more likely to elicit food-related vocabulary than a generic spatial patch.

This approach also revealed that the detector's training data becomes part of the captioning system's vocabulary. A detector trained on COCO categories can identify 80 object classes. A captioning model using those features can reliably describe those 80 categories but may struggle with objects outside the detector's vocabulary. This coupling between detector and captioner is both a strength (better object recognition) and a constraint (limited generalization).

Modern captioning systems are fully transformer-based. The image encoder is a Vision Transformer (ViT) or CLIP visual encoder, as we covered in the Vision Transformer chapter. The decoder is a standard transformer decoder that performs cross-attention over the visual token sequence.

The training objective remains the same autoregressive language modeling loss, but the architecture allows much richer cross-modal interaction. Every layer of the decoder can attend to every visual token, with different layers potentially specializing in different aspects: low-level appearance at early layers, object identity at intermediate layers, spatial relationships and scene-level context at later layers. This depth of interaction was not possible with the shallow single-step fusion of early CNN-LSTM systems.

BLIP-2 (Bootstrapping Language-Image Pre-training 2) represents a particularly clean example of modern captioning architecture. It uses a frozen image encoder (ViT-L from CLIP or EVA-CLIP) and a frozen language model (OPT or FlanT5), connected by a learned Q-Former module that extracts a small set of query vectors summarizing the visual content most relevant to language tasks. By keeping both the visual encoder and language model frozen, BLIP-2 can use powerful pretrained components without catastrophic forgetting, and the Q-Former learns to bridge the representation gap between them.

Automatic evaluation of captions is difficult. A caption can be perfectly accurate in different words than the reference. "A brown dog runs across a green field" and "A canine sprints through grass" describe the same image but share almost no vocabulary.

The standard evaluation metrics each capture different aspects:

• BLEU (Bilingual Evaluation Understudy): Measures n-gram overlap with reference captions. Fast and interpretable but penalizes valid paraphrases heavily.
• METEOR: Adds stemming and synonym matching alongside paraphrase matching on top of n-gram comparison. Better at catching semantic equivalence.
• CIDEr (Consensus-based Image Description Evaluation): Weights n-gram matches by their TF-IDF score across the full caption dataset. Penalizes generic phrases like "a man is" and rewards specific, information-rich content.
• SPICE (Semantic Propositional Image Caption Evaluation): Parses both generated and reference captions into scene graphs, then compares the graph structure. More reliable to paraphrase but computationally expensive.

No single metric captures all aspects of caption quality. In practice, CIDEr correlates best with human judgment on standard benchmarks, so it has become the primary optimization target in most captioning research. A concerning side effect of this is that models optimized directly for CIDEr, using reinforcement learning with CIDEr as the reward, produce captions that score well on the metric but strike human readers as formulaic or oddly specific in ways that match training data patterns rather than actual image content.

More recent work on caption evaluation uses large language models as judges. A judge model receives the generated caption, the reference captions, and sometimes the image itself, then rates its accuracy and fluency alongside informativeness on numerical scales. This approach correlates more strongly with human judgments than any single automatic metric, though it introduces its own biases depending on which language model is judge.

Visual grounding, sometimes called referring expression comprehension, asks a model to locate a specific object in an image given a natural language description. The input is an image plus a referring expression like "the woman in the red jacket on the left," and the output is a bounding box, segmentation mask, or spatial coordinates identifying the referred object.

Visual grounding requires understanding both visual attributes (colors, shapes, sizes) and spatial relationships (left, right, next to, behind). It also requires disambiguation: if there are three women in the image, "the woman in the red jacket" must specifically pick out the one wearing red, not any woman.

Grounding is the spatial backbone of visual understanding. Many downstream tasks depend on it. A model answering "Is the cat on the sofa or the floor?" must ground "the cat" and determine whether its location matches "on the sofa." A robot following instructions to "pick up the blue mug" must ground "the blue mug" to act on it. An image editing model instructed to "make the sky darker" must ground "the sky" before it can modify it.

The difference between VQA and grounding is the output format. VQA produces text. Grounding produces spatial coordinates or a pixel-level mask. This distinction matters architecturally: grounding models need either coordinate regression heads or dense spatial prediction architectures, not just language generation. You can think of grounding as VQA where the answer is spatial: the model must answer "where" rather than "what."

Grounding also enables compositionality in visual reasoning. By grounding each entity in a complex query to its image region, a system can verify relational claims: "Is the car in front of the truck?" can be answered by grounding "the car" and "the truck" separately and then computing their spatial relationship from their bounding boxes. This decomposition of visual reasoning into grounding plus spatial inference is more reliable than end-to-end approaches that try to answer the full question without intermediate localization.

The primary benchmark datasets for visual grounding are RefCOCO, RefCOCO+, and RefCOCOg. These datasets were collected using a reference game: two players see an image, one player selects an object, and the other player must describe it specifically enough that the first player can re-identify it from the description alone. This game-collection methodology produces natural, disambiguation-focused descriptions rather than generic labels.

RefCOCO and RefCOCO+ allow spatial language like "left," "right," "top," and "bottom," making them tests of spatial reasoning. RefCOCO+ disallows location words, forcing descriptions based purely on appearance. RefCOCOg uses longer, more complex descriptions collected from crowd workers rather than a two-player game, creating richer expressions that often describe relationships between multiple objects.

Two-stage grounding models first generate a set of candidate region proposals using an object detector, then score each region against the referring expression to find the best match. This pipeline is intuitive but slow: every candidate region must be scored against the text, and the model is fundamentally limited to regions the detector proposes. If the detector misses the referred object entirely, no amount of language understanding will recover it.

Single-stage grounding models directly predict bounding box coordinates from the joint image-text representation without an intermediate proposal step. Models like MDETR and TransVG use a transformer encoder to jointly encode image patches and text tokens, then add a bounding box prediction head that regresses to $(x_\text{center}, y_\text{center}, w, h)$ coordinates.

The regression target is typically normalized to $[0, 1]$ relative to image dimensions:

where each value is in the range $[0, 1]$ as a fraction of image width or height. Training uses a combination of L1 loss on the coordinates and the generalized IoU loss, which penalizes non-overlapping predictions more strongly than standard IoU:

where:

• $b$: ground truth bounding box coordinates, normalized to $[0, 1]$
• $\hat{b}$: predicted bounding box coordinates from the regression head
• $\mathcal{L}_\text{GIoU}$: generalized intersection-over-union loss, which accounts for the case where predicted and ground truth boxes do not overlap at all
• $\lambda_1, \lambda_2$: loss weighting hyperparameters, typically set to balance the two terms

The generalized IoU loss is necessary because standard IoU has zero gradient when predicted and ground truth boxes do not overlap. If a predicted box is placed in the wrong corner of the image, standard IoU provides no signal for which direction to move it. GIoU adds a term that penalizes the area of the smallest enclosing box that covers both prediction and target. This provides gradients even for non-overlapping boxes.

Visual grounding encompasses two related but distinct tasks. In referring expression comprehension, a single noun phrase uniquely identifies one object in the image, and the model must locate it. In phrase grounding, every noun phrase in a caption or description must be grounded to its corresponding region simultaneously.

Phrase grounding requires locating multiple expressions jointly, respecting their relationships. The phrase "the dog chasing the frisbee" requires grounding both "the dog" and "the frisbee," and their grounded locations should be spatially consistent with a chasing relationship. This is harder than independent phrase grounding because the spatial coherence between groundings must be maintained.

Dense captioning is a related task that inverts phrase grounding: rather than localizing phrases from a given caption, the model generates captions for detected regions. Given an image, a dense captioning model produces a set of (bounding box, caption) pairs describing each notable element in the scene. This is useful for generating richly annotated image descriptions that go beyond a single summary sentence.

Grounding models are evaluated using top-1 accuracy at different IoU thresholds. The standard threshold is IoU $\geq 0.5$: a prediction is correct if the predicted bounding box has at least 50% overlap with the ground truth box. Some benchmarks additionally report at IoU $\geq 0.25$ for lenient evaluation and IoU $\geq 0.75$ for strict evaluation.

The choice of threshold matters substantially for interpreting results. A model might achieve 85% accuracy at IoU $\geq 0.5$ but only 55% accuracy at IoU $\geq 0.75$ because many of its predictions are in approximately the right location but not precisely aligned with the object boundary. For applications like robotics where precise localization matters for manipulation, the stricter threshold is more informative.

Large VLMs like GPT-4V, Qwen-VL, and Grounding DINO extend grounding capability to instruction-following interfaces. You can ask "Where is the stop sign?" and receive a bounding box in the response. Grounding-capable VLMs output special tokens encoding spatial coordinates, which are decoded into box coordinates by the tokenizer.

Qwen-VL uses a particularly elegant approach: bounding boxes are represented as sequences of discrete coordinate tokens like <box>(0.23, 0.11), (0.67, 0.89)</box>. During training, the model learns to generate these coordinate token sequences alongside natural language answers. This unification means grounding is just another form of language generation, handled by the same autoregressive decoder.

This integration of grounding into general VLMs is significant because it means a single model can answer questions, generate captions, and locate objects, using the same parameters for all three tasks. The model does not need separate specialist components, separate training pipelines, or separate inference paths. Grounding is simply another mode of response, triggered by the phrasing of the input.

Scene understanding is the most complete image understanding task. Rather than asking one question, grounding one phrase, or generating one caption, scene understanding requires building a complete structural representation of the visual world: which objects are present, where they are, what properties they have, and how they relate to each other.

Full scene understanding typically includes:

• Object detection: Locate all objects with bounding boxes and class labels
• Instance segmentation: Assign each pixel to a specific object instance
• Semantic segmentation: Assign each pixel a semantic category label
• Relationship detection: Identify visual relationships between object pairs
• Scene graph generation: Build a structured graph encoding all entities and their relationships
• Depth estimation: Estimate the 3D distance of each pixel from the camera
• Scene categorization: Label the overall scene type (kitchen, street, park, etc.)

Not every application needs all of these. Autonomous driving primarily needs detection and segmentation alongside depth estimation. Image retrieval primarily needs object detection and scene categorization. Robotics needs all of them. The set of tasks you solve determines the architecture you build, though modern foundation model approaches increasingly target all subtasks with a single model.

A scene graph is the formal data structure for representing scene understanding outputs. Nodes are objects (entities), and directed edges are relationships between them. The scene graph for an image of "a person riding a bicycle on a road" would contain person, bicycle, road nodes, with edges: person -> bicycle (riding), bicycle -> road (on).

Scene graph generation models must detect objects, classify their relationships, and integrate the two. This is harder than it sounds because relationship labels depend on both the subject and object. "Behind" is a geometric relationship. "Wearing" describes physical attachment between a person and clothing. "Carrying" implies agency and physical support. "Same color as" is a cross-entity attribute comparison. A model must learn all of these relationship types and their visual signatures simultaneously.

The relationship vocabulary in scene graph datasets is long-tailed: a few common relationships like "on," "next to," and "has" appear in the large majority of training examples, while less common but informative relationships like "pouring," "leaning against," and "feeding" are rare. Models trained on these datasets tend to predict common relationships even when less common ones are correct, which limits the semantic richness of generated scene graphs.

The downstream utility of scene graphs extends to visual question answering. A VQA system that first generates a scene graph, then reasons over the graph structure to answer questions, can handle multi-hop relational questions that end-to-end neural models often fail on. "Is the thing being ridden in the photo the same color as the helmet?" requires identifying the bicycle, the helmet, and comparing their colors, an operation that is natural over a scene graph but harder for a model that must do everything implicitly.

Semantic segmentation assigns a class label to every pixel in the image. Every pixel belonging to a person gets the "person" label. Every pixel belonging to road surface gets the "road" label. The model produces a dense prediction map with the same spatial resolution as the input image, where each position holds a class probability distribution.

Instance segmentation goes further: it distinguishes individual instances of the same class. Where semantic segmentation labels all car pixels as "car," instance segmentation labels the pixels of the leftmost car as "car #1" and the pixels of the rightmost car as "car #2." This distinction is important for understanding scenes with multiple objects of the same type.

The Mask R-CNN architecture (He et al., 2017) established the standard approach for instance segmentation. Building on Faster R-CNN for object detection, Mask R-CNN adds a parallel mask prediction branch that outputs a small binary mask for each detected bounding box. For each region proposal, the network predicts both the object class and a $28 \times 28$ binary mask showing which pixels within the proposal belong to the object. The mask is resized to match the proposal dimensions during inference.

Panoptic segmentation unifies semantic and instance segmentation. In semantic segmentation, every pixel gets a class label, but multiple instances of the same class share one label: all pixels belonging to any car are labeled "car." In instance segmentation, each individual object instance gets a unique label, but background regions (sky, road, grass) are often treated separately.

Panoptic segmentation requires both: every pixel is assigned both a semantic class label and an instance ID. "Stuff" categories (sky, road, water) have one instance each, while "thing" categories (person, car, dog) have separate instance IDs per object.

The panoptic quality (PQ) metric evaluates panoptic segmentation:

where:

• $\text{TP}$: true positive matches between predicted and ground truth segments
• $\text{FP}$: predicted segments with no matching ground truth segment
• $\text{FN}$: ground truth segments with no matching predicted segment
• $\text{IoU}(p, g)$: intersection over union between matched prediction $p$ and ground truth $g$

This metric rewards both detection accuracy, captured by the TP/FP/FN counts, and segmentation quality, captured by the IoU term for matched pairs. A model that detects all objects but segments them imprecisely will have high recall but low average IoU, creating a moderate PQ score. Getting high PQ requires both finding all objects and precisely delineating their boundaries.

Panoptic segmentation can be decomposed into PQ = SQ $\times$ RQ, where SQ is the segmentation quality (average IoU over matched pairs) and RQ is the recognition quality (the F1 score over matching with IoU $> 0.5$). This decomposition lets you diagnose whether performance limitations come from imprecise segmentation or from missing detections.

Depth estimation adds a third dimension to scene understanding. For each pixel, the model estimates its distance from the camera, creating a depth map that encodes the three-dimensional structure of the scene.

Monocular depth estimation, which infers depth from a single image without a stereo pair or active sensor, is fundamentally an ill-posed problem: any image is compatible with infinitely many 3D scenes. Models must use visual cues like perspective, occlusion, relative size, texture gradients, and defocus blur to estimate depth. These cues are strong in natural images because the world follows consistent geometric rules, but they are ambiguous in unusual scenes or synthetic images.

Early approaches used encoder-decoder architectures where the encoder extracts high-level features and the decoder upsamples them while incorporating skip connections to preserve spatial detail. DPT (Dense Prediction Transformer, Ranftl et al., 2021) demonstrated that vision transformers substantially outperform convolution-based encoders for depth estimation, because the global receptive field of self-attention helps maintain depth consistency across the full image rather than only using local context.

Traditional scene understanding required training separate specialist models for each subtask. Foundation models change this by giving generalist scene understanding through prompting.

The Segment Anything Model (SAM) from Meta demonstrates that a single model pretrained on an enormous collection of diverse segmentation data can segment any object in any image given a prompt. The prompt can be a point click, a bounding box, or a text description, and SAM generates a segmentation mask for the specified region. SAM's generality reflects more than convenience: the model has learned a broad concept of "objectness" that transfers to categories it has never been explicitly trained to segment, including unusual objects and art from domains far from its training distribution.

GLIP and Grounding DINO extend this to language-driven detection and scene understanding. Rather than detecting a fixed set of object classes, these models detect any object described in natural language. Ask "find all red vehicles" and the model returns bounding boxes for objects matching that description, even if "red vehicle" was never an explicit training category. This open-vocabulary detection is enabled by aligning visual region features with language features from a pretrained encoder, so that region features matching "red vehicle" are close to the text embedding of "red vehicle" in a shared representation space.

The trend toward foundation models for scene understanding reflects a deeper insight: the richness of natural language supervision provides a more complete training signal than manually defined class vocabularies. A model trained to align visual regions with arbitrary text descriptions implicitly learns a much broader and more fine-grained concept space than a model trained only to classify among 80 COCO categories.

To see how these capabilities work in practice, let us build a pipeline that demonstrates VQA and captioning alongside grounding using modern pretrained models. We will use the Transformers library from Hugging Face, which provides easy access to BLIP and other vision-language models. The BLIP model family (Salesforce) is particularly well-suited for teaching because it cleanly separates captioning and VQA variants while sharing the same underlying architecture.

First, we install the required packages and prepare a demo image to work with throughout the tutorial.

The image is a standard COCO validation photo showing two cats lounging on a pink couch. We will use this same image across all three tasks to compare how each model interprets the same visual content. Fixing the image lets us see directly what changes: the model architecture, the task formulation, and the output format.

BLIP (Bootstrapping Language-Image Pre-training) is a versatile vision-language model that handles both captioning and VQA through the same underlying encoder but task-specific decoder heads. Let us load the captioning variant and generate descriptions.

BLIP's captioning model uses a visual transformer encoder paired with a BERT-style language decoder that has been adapted for autoregressive generation. During pretraining, BLIP jointly trains on three objectives: image-text contrastive learning (like CLIP), image-text matching classification, and caption generation. This multi-task pretraining produces representations that are useful for many downstream tasks.

The conditional caption demonstrates how a prompt prefix can steer the generation style. The unconditional version describes the scene in whatever format the model learned during pretraining, while the conditional version constrains it to start with "a photo of..." This is analogous to prompted generation in language models: the prefix conditions the entire sequence that follows.

BLIP also includes a VQA variant fine-tuned on VQA datasets. The same visual encoder is used, but the decoder is adapted to treat the question as a conditional input and generate a short answer. We will ask several questions covering different reasoning types to see how well the model handles counting, attribute recognition, and scene-level inference.

Notice how the model handles different question types. Counting questions require the model to enumerate objects. Attribute questions require it to identify the property of a specific entity. Scene-level inference questions require integrating cues such as furniture and lighting with the environment type. Each requires attending to different image regions and applying different reasoning patterns, yet all are handled by the same model through the same generation mechanism.

One of the key decisions in caption generation is the decoding strategy. Beam search finds the most probable sequence according to the model's learned distribution, while sampling introduces diversity by stochastically selecting from the probability distribution at each step. Understanding the trade-off matters for production applications where either reliability or variety is preferred.

Beam search produces deterministic outputs that become slightly more fluent as beam width increases, because a wider beam explores more candidate sequences before committing. Sampling introduces variety at the cost of occasional lower-quality outputs. Production systems often use beam search for reliability, while diverse caption generation for creative applications or data augmentation might prefer sampling.

One way to quantify how different the generated captions are from each other is to compute n-gram overlap scores. If beam search captions are all very similar to each other and sampling captions are more diverse, we expect beam captions to have higher mutual consistency. Let us measure this directly.

To understand the diversity of reasoning required in VQA benchmarks, let us visualize the distribution of answer types in a representative dataset. This distribution reveals both the composition of typical VQA benchmarks and the potential failure modes that arise from class imbalance.

The dominance of Yes/No questions changes the interpretation for model evaluation. A model that simply answers "yes" to every question would achieve roughly 38% accuracy without understanding any image. This is why balanced datasets like VQA v2 pair complementary images to make shortcut answers impossible. Even so, Yes/No questions remain harder to use as a true test of visual reasoning because the binary answer space can still be gamed by other statistical patterns.

One of the most interpretable aspects of modern VQA models is their attention maps. These maps show which image regions the model focuses on when generating each answer. Different question types naturally direct attention to different regions: counting questions concentrate on individual object locations, while attribute questions spread attention more broadly across the relevant region.

The counting question focuses on individual object locations, while the color question distributes attention more broadly across the couch region. This is the cross-modal attention mechanism at work: the question text steers which parts of the visual representation get emphasized when creating the answer. In real transformer-based VQA models, multiple attention heads perform this kind of selection in parallel, with different heads potentially specializing in different aspects of the visual scene.

Different evaluation metrics capture different aspects of caption quality. BLEU measures exact n-gram overlap, which penalizes valid paraphrases. CIDEr rewards content-specific vocabulary. Understanding how these metrics behave on the same captions reveals why no single metric is a perfect proxy for caption quality.

The metric comparison reveals an important tension in captioning evaluation: BLEU-4 is very sensitive to exact phrase matches, so short or differently-worded captions score poorly even when they accurately describe the image. CIDEr-like scores reward captions that use content-specific vocabulary shared with multiple references, making the metric more reliable to paraphrases that use the same specific words. Neither metric perfectly captures whether a caption is accurate and informative; they each capture a different facet of textual similarity.

Image understanding systems have achieved remarkable performance on standard benchmarks, but a candid assessment reveals important gaps between benchmark performance and visual understanding. Understanding these gaps is needed for deploying these systems responsibly.

The most persistent issue is compositional reasoning. Models frequently fail on questions that require combining multiple visual concepts in ways not well-represented during training. "Is the object to the left of the person also smaller than the object behind the person?" requires tracking multiple spatial relationships simultaneously and comparing them. Current models, including large VLMs, handle this kind of compositional reasoning much less reliably than simple single-step questions. The problem is that benchmark datasets often contain shortcut correlations: the most common color of objects labeled "stop sign" is always red, so a model can answer "What color is the stop sign?" correctly without reading the sign's color from the image.

This shortcut learning problem is structural, not a matter of model capacity. Adding more parameters does not eliminate the shortcuts; it can even make them worse by giving the model more capacity to memorize training statistics. Addressing it requires careful dataset design (like VQA v2's complementary pairs), adversarial evaluation (like VQA-CP), and training techniques specifically designed to prevent statistical shortcut exploitation.

Language priors represent another source of benchmark inflation. VQA models sometimes learn to associate question patterns with likely answers regardless of image content. "Is the sky blue?" is likely answered "yes" because skies are blue in most training images. "What do you use to cut vegetables?" will often be answered "knife" based on language co-occurrence statistics, even when the relevant image shows scissors. Models trained predominantly on English-language web data inherit all the statistical regularities of that corpus. High VQA accuracy can mask visual reasoning deficits that only become apparent on adversarial or out-of-distribution questions.

Grounding and captioning face a different but equally serious limitation: hallucination. Models confidently describe objects that are not in the image, assert relationships that do not hold, or generate plausible-sounding but incorrect attribute descriptions. Language model priors dominate when the visual signal is ambiguous or the scene is unusual. A model trained on many images of kitchens may describe a kitchen even when shown a bathroom with similar colors and spatial layout. This hallucination problem is particularly acute in medical imaging applications: a system that confidently reports findings not present in a scan has failed in the most dangerous possible way, regardless of its benchmark accuracy.

Evaluation metrics remain a bottleneck for measuring real progress. BLEU, METEOR, CIDEr measure surface form overlap rather than semantic accuracy. A caption can score poorly on all three metrics while being perfectly accurate and well-written, simply because it uses different but correct words than the reference captions. Human evaluation correlates more reliably with actual quality but is expensive and not reproducible at scale. The field is moving toward using large language models as automated judges, which improves correlation with human judgments but introduces new concerns about judge bias and the circularity of using one language model to evaluate another.

Computational cost remains a practical barrier for many deployment scenarios. Models like BLIP-2 and GPT-4V are far too large for edge deployment on smartphones, embedded systems, or in privacy-constrained environments where data cannot leave the device. Efficient vision-language models like MobileVLM and TinyLLaVA address this through architectural simplification, knowledge distillation, and quantization, but there is still a substantial performance gap between efficient and state-of-the-art models. As vision-language models become more deeply integrated into consumer products, the efficiency gap will become more commercially important.

Cultural and demographic biases embedded in training data propagate into image understanding outputs in ways that are difficult to detect and correct. Caption generation models produce lower quality captions for images from geographic regions and cultural contexts underrepresented in training data. VQA systems perform worse on questions about cultural practices and foods as well as objects from underrepresented demographics. Object detectors have higher false negative rates for images of darker-skinned individuals due to inadequate representation in benchmark datasets. These biases are not simply fixable by scaling: larger models trained on more web data may amplify biases present in that data rather than correcting them. Addressing these issues requires deliberate diversity in data collection, targeted evaluation on underrepresented subgroups, and ongoing monitoring in deployment.

Finally, the evaluation of scene understanding under distribution shift deserves attention. Models that achieve high performance on clean, well-lit, frontally-oriented images can fail dramatically on the kinds of images encountered in real deployments: blurry, low-light, partially occluded, or from unusual viewpoints. This sensitivity to input distribution is a basic characteristic of models trained primarily on curated benchmark datasets.

Image understanding covers four interrelated tasks that together test a model's capacity to reason jointly over visual and textual information.

Visual question answering evaluates targeted visual reasoning by asking natural language questions and scoring the answers. It has evolved from closed-vocabulary classification using CNN-extracted features to open-ended generation with large vision-language models, with cross-modal attention letting the model to focus on image regions relevant to each specific question. The benchmark history of VQA reveals how easily models exploit dataset statistics, motivating careful dataset design and adversarial evaluation.

Image captioning generates descriptive text without a guiding question, requiring the model to independently determine what aspects of a scene deserve description and to express them in fluent natural language. The progression from fixed-vector CNN-LSTM systems to spatial attention to region-based features to fully transformer-based architectures each addressed specific limitations of the previous approach. Evaluation using metrics like CIDEr and SPICE captures different aspects of quality but none fully replaces human judgment.

Visual grounding maps natural language phrases to spatial locations in images. This provides the spatial precision needed for downstream tasks like robotics and interactive editing. Single-stage grounding models trained with L1 and GIoU losses can directly regress bounding box coordinates from joint image-text representations. The integration of grounding into instruction-following VLMs means spatial localization is now just another output format rather than a separate capability requiring a separate model.

Scene understanding goes furthest, building complete structural representations of the visual world through panoptic segmentation, scene graph generation, depth estimation, and scene categorization. Foundation models like SAM and Grounding DINO demonstrate that open-vocabulary, generalist scene understanding is achievable through language supervision, replacing the specialist pipelines that previously dominated the field.

The remaining challenges center on compositional reasoning, hallucination, reliable evaluation, computational efficiency, and demographic bias. These are not minor technical issues to be patched in the next version: they represent basic questions about what visual understanding means and how to measure it accurately. As we explore image generation in the next chapter, these same limitations will appear in a generative context, where the model must understand images and create them to satisfy complex textual specifications.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about image understanding tasks and the models that tackle them.