Image Understanding › Scene Understanding

Scene understanding is the most complete image understanding task. Rather than asking one question, grounding one phrase, or generating one caption, scene understanding requires building a complete structural representation of the visual world: which objects are present, where they are, what properties they have, and how they relate to each other.

Full scene understanding typically includes:

• Object detection: Locate all objects with bounding boxes and class labels
• Instance segmentation: Assign each pixel to a specific object instance
• Semantic segmentation: Assign each pixel a semantic category label
• Relationship detection: Identify visual relationships between object pairs
• Scene graph generation: Build a structured graph encoding all entities and their relationships
• Depth estimation: Estimate the 3D distance of each pixel from the camera
• Scene categorization: Label the overall scene type (kitchen, street, park, etc.)

Not every application needs all of these. Autonomous driving primarily needs detection and segmentation alongside depth estimation. Image retrieval primarily needs object detection and scene categorization. Robotics needs all of them. The set of tasks you solve determines the architecture you build, though modern foundation model approaches increasingly target all subtasks with a single model.

A scene graph is the formal data structure for representing scene understanding outputs. Nodes are objects (entities), and directed edges are relationships between them. The scene graph for an image of "a person riding a bicycle on a road" would contain person, bicycle, road nodes, with edges: person -> bicycle (riding), bicycle -> road (on).

Scene graph generation models must detect objects, classify their relationships, and integrate the two. This is harder than it sounds because relationship labels depend on both the subject and object. "Behind" is a geometric relationship. "Wearing" describes physical attachment between a person and clothing. "Carrying" implies agency and physical support. "Same color as" is a cross-entity attribute comparison. A model must learn all of these relationship types and their visual signatures simultaneously.

The relationship vocabulary in scene graph datasets is long-tailed: a few common relationships like "on," "next to," and "has" appear in the large majority of training examples, while less common but informative relationships like "pouring," "leaning against," and "feeding" are rare. Models trained on these datasets tend to predict common relationships even when less common ones are correct, which limits the semantic richness of generated scene graphs.

The downstream utility of scene graphs extends to visual question answering. A VQA system that first generates a scene graph, then reasons over the graph structure to answer questions, can handle multi-hop relational questions that end-to-end neural models often fail on. "Is the thing being ridden in the photo the same color as the helmet?" requires identifying the bicycle, the helmet, and comparing their colors, an operation that is natural over a scene graph but harder for a model that must do everything implicitly.

Semantic segmentation assigns a class label to every pixel in the image. Every pixel belonging to a person gets the "person" label. Every pixel belonging to road surface gets the "road" label. The model produces a dense prediction map with the same spatial resolution as the input image, where each position holds a class probability distribution.

Instance segmentation goes further: it distinguishes individual instances of the same class. Where semantic segmentation labels all car pixels as "car," instance segmentation labels the pixels of the leftmost car as "car #1" and the pixels of the rightmost car as "car #2." This distinction is important for understanding scenes with multiple objects of the same type.

The Mask R-CNN architecture (He et al., 2017) established the standard approach for instance segmentation. Building on Faster R-CNN for object detection, Mask R-CNN adds a parallel mask prediction branch that outputs a small binary mask for each detected bounding box. For each region proposal, the network predicts both the object class and a $28 \times 28$ binary mask showing which pixels within the proposal belong to the object. The mask is resized to match the proposal dimensions during inference.

Panoptic segmentation unifies semantic and instance segmentation. In semantic segmentation, every pixel gets a class label, but multiple instances of the same class share one label: all pixels belonging to any car are labeled "car." In instance segmentation, each individual object instance gets a unique label, but background regions (sky, road, grass) are often treated separately.

Panoptic segmentation requires both: every pixel is assigned both a semantic class label and an instance ID. "Stuff" categories (sky, road, water) have one instance each, while "thing" categories (person, car, dog) have separate instance IDs per object.

The panoptic quality (PQ) metric evaluates panoptic segmentation:

$$\text{PQ} = \frac{\sum_{(p, g) \in \text{TP}} \text{IoU}(p, g)}{|\text{TP}| + \frac{1}{2}|\text{FP}| + \frac{1}{2}|\text{FN}|}$$

where:

• $\text{TP}$: true positive matches between predicted and ground truth segments
• $\text{FP}$: predicted segments with no matching ground truth segment
• $\text{FN}$: ground truth segments with no matching predicted segment
• $\text{IoU}(p, g)$: intersection over union between matched prediction $p$ and ground truth $g$

This metric rewards both detection accuracy, captured by the TP/FP/FN counts, and segmentation quality, captured by the IoU term for matched pairs. A model that detects all objects but segments them imprecisely will have high recall but low average IoU, creating a moderate PQ score. Getting high PQ requires both finding all objects and precisely delineating their boundaries.

Panoptic segmentation can be decomposed into PQ = SQ $\times$ RQ, where SQ is the segmentation quality (average IoU over matched pairs) and RQ is the recognition quality (the F1 score over matching with IoU $> 0.5$). This decomposition lets you diagnose whether performance limitations come from imprecise segmentation or from missing detections.

Depth estimation adds a third dimension to scene understanding. For each pixel, the model estimates its distance from the camera, creating a depth map that encodes the three-dimensional structure of the scene.

Monocular depth estimation, which infers depth from a single image without a stereo pair or active sensor, is fundamentally an ill-posed problem: any image is compatible with infinitely many 3D scenes. Models must use visual cues like perspective, occlusion, relative size, texture gradients, and defocus blur to estimate depth. These cues are strong in natural images because the world follows consistent geometric rules, but they are ambiguous in unusual scenes or synthetic images.

Early approaches used encoder-decoder architectures where the encoder extracts high-level features and the decoder upsamples them while incorporating skip connections to preserve spatial detail. DPT (Dense Prediction Transformer, Ranftl et al., 2021) demonstrated that vision transformers substantially outperform convolution-based encoders for depth estimation, because the global receptive field of self-attention helps maintain depth consistency across the full image rather than only using local context.

Traditional scene understanding required training separate specialist models for each subtask. Foundation models change this by giving generalist scene understanding through prompting.

The Segment Anything Model (SAM) from Meta demonstrates that a single model pretrained on an enormous collection of diverse segmentation data can segment any object in any image given a prompt. The prompt can be a point click, a bounding box, or a text description, and SAM generates a segmentation mask for the specified region. SAM's generality reflects more than convenience: the model has learned a broad concept of "objectness" that transfers to categories it has never been explicitly trained to segment, including unusual objects and art from domains far from its training distribution.

GLIP and Grounding DINO extend this to language-driven detection and scene understanding. Rather than detecting a fixed set of object classes, these models detect any object described in natural language. Ask "find all red vehicles" and the model returns bounding boxes for objects matching that description, even if "red vehicle" was never an explicit training category. This open-vocabulary detection is enabled by aligning visual region features with language features from a pretrained encoder, so that region features matching "red vehicle" are close to the text embedding of "red vehicle" in a shared representation space.

The trend toward foundation models for scene understanding reflects a deeper insight: the richness of natural language supervision provides a more complete training signal than manually defined class vocabularies. A model trained to align visual regions with arbitrary text descriptions implicitly learns a much broader and more fine-grained concept space than a model trained only to classify among 80 COCO categories.