Image Understanding › Visual Grounding

Visual grounding, sometimes called referring expression comprehension, asks a model to locate a specific object in an image given a natural language description. The input is an image plus a referring expression like "the woman in the red jacket on the left," and the output is a bounding box, segmentation mask, or spatial coordinates identifying the referred object.

Visual grounding requires understanding both visual attributes (colors, shapes, sizes) and spatial relationships (left, right, next to, behind). It also requires disambiguation: if there are three women in the image, "the woman in the red jacket" must specifically pick out the one wearing red, not any woman.

Grounding is the spatial backbone of visual understanding. Many downstream tasks depend on it. A model answering "Is the cat on the sofa or the floor?" must ground "the cat" and determine whether its location matches "on the sofa." A robot following instructions to "pick up the blue mug" must ground "the blue mug" to act on it. An image editing model instructed to "make the sky darker" must ground "the sky" before it can modify it.

The difference between VQA and grounding is the output format. VQA produces text. Grounding produces spatial coordinates or a pixel-level mask. This distinction matters architecturally: grounding models need either coordinate regression heads or dense spatial prediction architectures, not just language generation. You can think of grounding as VQA where the answer is spatial: the model must answer "where" rather than "what."

Grounding also enables compositionality in visual reasoning. By grounding each entity in a complex query to its image region, a system can verify relational claims: "Is the car in front of the truck?" can be answered by grounding "the car" and "the truck" separately and then computing their spatial relationship from their bounding boxes. This decomposition of visual reasoning into grounding plus spatial inference is more reliable than end-to-end approaches that try to answer the full question without intermediate localization.

The primary benchmark datasets for visual grounding are RefCOCO, RefCOCO+, and RefCOCOg. These datasets were collected using a reference game: two players see an image, one player selects an object, and the other player must describe it specifically enough that the first player can re-identify it from the description alone. This game-collection methodology produces natural, disambiguation-focused descriptions rather than generic labels.

RefCOCO and RefCOCO+ allow spatial language like "left," "right," "top," and "bottom," making them tests of spatial reasoning. RefCOCO+ disallows location words, forcing descriptions based purely on appearance. RefCOCOg uses longer, more complex descriptions collected from crowd workers rather than a two-player game, creating richer expressions that often describe relationships between multiple objects.

Two-stage grounding models first generate a set of candidate region proposals using an object detector, then score each region against the referring expression to find the best match. This pipeline is intuitive but slow: every candidate region must be scored against the text, and the model is fundamentally limited to regions the detector proposes. If the detector misses the referred object entirely, no amount of language understanding will recover it.

Single-stage grounding models directly predict bounding box coordinates from the joint image-text representation without an intermediate proposal step. Models like MDETR and TransVG use a transformer encoder to jointly encode image patches and text tokens, then add a bounding box prediction head that regresses to $(x_\text{center}, y_\text{center}, w, h)$ coordinates.

The regression target is typically normalized to $[0, 1]$ relative to image dimensions:

$$\hat{b} = (\hat{x}_\text{center}, \hat{y}_\text{center}, \hat{w}, \hat{h})$$

where each value is in the range $[0, 1]$ as a fraction of image width or height. Training uses a combination of L1 loss on the coordinates and the generalized IoU loss, which penalizes non-overlapping predictions more strongly than standard IoU:

$$\mathcal{L}_\text{ground} = \lambda_1 \|b - \hat{b}\|_1 + \lambda_2 \mathcal{L}_\text{GIoU}(b, \hat{b})$$

where:

• $b$: ground truth bounding box coordinates, normalized to $[0, 1]$
• $\hat{b}$: predicted bounding box coordinates from the regression head
• $\mathcal{L}_\text{GIoU}$: generalized intersection-over-union loss, which accounts for the case where predicted and ground truth boxes do not overlap at all
• $\lambda_1, \lambda_2$: loss weighting hyperparameters, typically set to balance the two terms

The generalized IoU loss is necessary because standard IoU has zero gradient when predicted and ground truth boxes do not overlap. If a predicted box is placed in the wrong corner of the image, standard IoU provides no signal for which direction to move it. GIoU adds a term that penalizes the area of the smallest enclosing box that covers both prediction and target. This provides gradients even for non-overlapping boxes.

Visual grounding encompasses two related but distinct tasks. In referring expression comprehension, a single noun phrase uniquely identifies one object in the image, and the model must locate it. In phrase grounding, every noun phrase in a caption or description must be grounded to its corresponding region simultaneously.

Phrase grounding requires locating multiple expressions jointly, respecting their relationships. The phrase "the dog chasing the frisbee" requires grounding both "the dog" and "the frisbee," and their grounded locations should be spatially consistent with a chasing relationship. This is harder than independent phrase grounding because the spatial coherence between groundings must be maintained.

Dense captioning is a related task that inverts phrase grounding: rather than localizing phrases from a given caption, the model generates captions for detected regions. Given an image, a dense captioning model produces a set of (bounding box, caption) pairs describing each notable element in the scene. This is useful for generating richly annotated image descriptions that go beyond a single summary sentence.

Grounding models are evaluated using top-1 accuracy at different IoU thresholds. The standard threshold is IoU $\geq 0.5$: a prediction is correct if the predicted bounding box has at least 50% overlap with the ground truth box. Some benchmarks additionally report at IoU $\geq 0.25$ for lenient evaluation and IoU $\geq 0.75$ for strict evaluation.

The choice of threshold matters substantially for interpreting results. A model might achieve 85% accuracy at IoU $\geq 0.5$ but only 55% accuracy at IoU $\geq 0.75$ because many of its predictions are in approximately the right location but not precisely aligned with the object boundary. For applications like robotics where precise localization matters for manipulation, the stricter threshold is more informative.

Large VLMs like GPT-4V, Qwen-VL, and Grounding DINO extend grounding capability to instruction-following interfaces. You can ask "Where is the stop sign?" and receive a bounding box in the response. Grounding-capable VLMs output special tokens encoding spatial coordinates, which are decoded into box coordinates by the tokenizer.

Qwen-VL uses a particularly elegant approach: bounding boxes are represented as sequences of discrete coordinate tokens like <box>(0.23, 0.11), (0.67, 0.89)</box>. During training, the model learns to generate these coordinate token sequences alongside natural language answers. This unification means grounding is just another form of language generation, handled by the same autoregressive decoder.

This integration of grounding into general VLMs is significant because it means a single model can answer questions, generate captions, and locate objects, using the same parameters for all three tasks. The model does not need separate specialist components, separate training pipelines, or separate inference paths. Grounding is simply another mode of response, triggered by the phrasing of the input.