Image Understanding › Summary

Image understanding covers four interrelated tasks that together test a model's capacity to reason jointly over visual and textual information.

Visual question answering evaluates targeted visual reasoning by asking natural language questions and scoring the answers. It has evolved from closed-vocabulary classification using CNN-extracted features to open-ended generation with large vision-language models, with cross-modal attention letting the model to focus on image regions relevant to each specific question. The benchmark history of VQA reveals how easily models exploit dataset statistics, motivating careful dataset design and adversarial evaluation.

Image captioning generates descriptive text without a guiding question, requiring the model to independently determine what aspects of a scene deserve description and to express them in fluent natural language. The progression from fixed-vector CNN-LSTM systems to spatial attention to region-based features to fully transformer-based architectures each addressed specific limitations of the previous approach. Evaluation using metrics like CIDEr and SPICE captures different aspects of quality but none fully replaces human judgment.

Visual grounding maps natural language phrases to spatial locations in images. This provides the spatial precision needed for downstream tasks like robotics and interactive editing. Single-stage grounding models trained with L1 and GIoU losses can directly regress bounding box coordinates from joint image-text representations. The integration of grounding into instruction-following VLMs means spatial localization is now just another output format rather than a separate capability requiring a separate model.

Scene understanding goes furthest, building complete structural representations of the visual world through panoptic segmentation, scene graph generation, depth estimation, and scene categorization. Foundation models like SAM and Grounding DINO demonstrate that open-vocabulary, generalist scene understanding is achievable through language supervision, replacing the specialist pipelines that previously dominated the field.

The remaining challenges center on compositional reasoning, hallucination, reliable evaluation, computational efficiency, and demographic bias. These are not minor technical issues to be patched in the next version: they represent basic questions about what visual understanding means and how to measure it accurately. As we explore image generation in the next chapter, these same limitations will appear in a generative context, where the model must understand images and create them to satisfy complex textual specifications.