Image Understanding › Implementation: Building an Image Understanding Pipeline
Language AI HandbookAll chapters Image Understanding › Implementation: Building an Image Understanding PipelineOne section · Sections in this chapter6 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To see how these capabilities work in practice, let us build a pipeline that demonstrates VQA and captioning alongside grounding using modern pretrained models. We will use the Transformers library from Hugging Face, which provides easy access to BLIP and other vision-language models. The BLIP model family (Salesforce) is particularly well-suited for teaching because it cleanly separates captioning and VQA variants while sharing the same underlying architecture.
Setup and Dependencies
First, we install the required packages and prepare a demo image to work with throughout the tutorial.
import subprocess
subprocess.run(
[
"uv",
"pip",
"install",
"transformers",
"Pillow",
"requests",
"torch",
"torchvision",
],
capture_output=True,
)from io import BytesIO
import requests
from PIL import Image
# Load a demo image from COCO validation set (two cats on a couch)
IMAGE_URL = "http://images.cocodataset.org/val2017/000000039769.jpg"
response = requests.get(IMAGE_URL, timeout=10)
image = Image.open(BytesIO(response.content)).convert("RGB")
image_width, image_height = image.sizeImage size: 640 x 480 pixels Image mode: RGB
The image is a standard COCO validation photo showing two cats lounging on a pink couch. We will use this same image across all three tasks to compare how each model interprets the same visual content. Fixing the image lets us see directly what changes: the model architecture, the task formulation, and the output format.
Image Captioning with BLIP
BLIP (Bootstrapping Language-Image Pre-training) is a versatile vision-language model that handles both captioning and VQA through the same underlying encoder but task-specific decoder heads. Let us load the captioning variant and generate descriptions.
BLIP's captioning model uses a visual transformer encoder paired with a BERT-style language decoder that has been adapted for autoregressive generation. During pretraining, BLIP jointly trains on three objectives: image-text contrastive learning (like CLIP), image-text matching classification, and caption generation. This multi-task pretraining produces representations that are useful for many downstream tasks.
from transformers import BlipForConditionalGeneration, BlipProcessor
caption_processor = BlipProcessor.from_pretrained(
"Salesforce/blip-image-captioning-base"
)
caption_model = BlipForConditionalGeneration.from_pretrained(
"Salesforce/blip-image-captioning-base"
)
caption_model.eval()import torch
# Unconditional captioning: describe the image without a guiding text prefix
with torch.no_grad():
inputs_uncond = caption_processor(image, return_tensors="pt")
ids_uncond = caption_model.generate(
**inputs_uncond, max_new_tokens=50, num_beams=5
)
caption_uncond = caption_processor.decode(
ids_uncond[0], skip_special_tokens=True
)
# Conditional captioning: steer the generation style with a prefix
inputs_cond = caption_processor(image, "a photo of", return_tensors="pt")
ids_cond = caption_model.generate(
**inputs_cond, max_new_tokens=50, num_beams=5
)
caption_cond = caption_processor.decode(
ids_cond[0], skip_special_tokens=True
)Unconditional caption: two cats sleeping on a couch Conditional caption (prompted with 'a photo of'): a photo of two cats sleeping on a couch
The conditional caption demonstrates how a prompt prefix can steer the generation style. The unconditional version describes the scene in whatever format the model learned during pretraining, while the conditional version constrains it to start with "a photo of..." This is analogous to prompted generation in language models: the prefix conditions the entire sequence that follows.
Visual Question Answering with BLIP
BLIP also includes a VQA variant fine-tuned on VQA datasets. The same visual encoder is used, but the decoder is adapted to treat the question as a conditional input and generate a short answer. We will ask several questions covering different reasoning types to see how well the model handles counting, attribute recognition, and scene-level inference.
from transformers import BlipForQuestionAnswering
vqa_processor = BlipProcessor.from_pretrained("Salesforce/blip-vqa-base")
vqa_model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
vqa_model.eval()questions = [
"What animals are in this image?",
"How many cats can you see?",
"What are the cats doing?",
"What is the color of the couch?",
"Is this indoors or outdoors?",
]
vqa_results = []
with torch.no_grad():
for question in questions:
inputs = vqa_processor(image, question, return_tensors="pt")
ids = vqa_model.generate(**inputs, max_new_tokens=20)
answer = vqa_processor.decode(ids[0], skip_special_tokens=True)
vqa_results.append((question, answer))Visual Question Answering Results: ------------------------------------------------------- Q: What animals are in this image? A: cats Q: How many cats can you see? A: 2 Q: What are the cats doing? A: sleeping Q: What is the color of the couch? A: pink Q: Is this indoors or outdoors? A: indoors
Notice how the model handles different question types. Counting questions require the model to enumerate objects. Attribute questions require it to identify the property of a specific entity. Scene-level inference questions require integrating cues such as furniture and lighting with the environment type. Each requires attending to different image regions and applying different reasoning patterns, yet all are handled by the same model through the same generation mechanism.
Caption Diversity: Beam Search vs. Sampling
One of the key decisions in caption generation is the decoding strategy. Beam search finds the most probable sequence according to the model's learned distribution, while sampling introduces diversity by stochastically selecting from the probability distribution at each step. Understanding the trade-off matters for production applications where either reliability or variety is preferred.
beam_captions = {}
sampling_captions = []
with torch.no_grad():
inputs = caption_processor(image, return_tensors="pt")
for n_beams in [1, 3, 5]:
ids = caption_model.generate(
**inputs, num_beams=n_beams, max_new_tokens=50
)
beam_captions[n_beams] = caption_processor.decode(
ids[0], skip_special_tokens=True
)
for _ in range(3):
ids = caption_model.generate(
**inputs,
do_sample=True,
temperature=0.9,
top_p=0.9,
max_new_tokens=50,
)
sampling_captions.append(
caption_processor.decode(ids[0], skip_special_tokens=True)
)Beam search captions (different beam widths): Beams=1: two cats sleeping on a couch Beams=3: two cats sleeping on a couch Beams=5: two cats sleeping on a couch Sampled captions (temperature=0.9, top_p=0.9): Sample 1: two cats sleeping Sample 2: two cats sleeping on a couch Sample 3: two cats laying next to each other
Beam search produces deterministic outputs that become slightly more fluent as beam width increases, because a wider beam explores more candidate sequences before committing. Sampling introduces variety at the cost of occasional lower-quality outputs. Production systems often use beam search for reliability, while diverse caption generation for creative applications or data augmentation might prefer sampling.
BLEU Score Consistency Analysis
One way to quantify how different the generated captions are from each other is to compute n-gram overlap scores. If beam search captions are all very similar to each other and sampling captions are more diverse, we expect beam captions to have higher mutual consistency. Let us measure this directly.
import math
from collections import Counter
def compute_ngrams(text, n):
tokens = text.lower().split()
return Counter(
[tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)]
)
def bleu_precision(candidate, references, n):
candidate_ngrams = compute_ngrams(candidate, n)
max_ref_counts = Counter()
for ref in references:
ref_ngrams = compute_ngrams(ref, n)
for ngram in candidate_ngrams:
max_ref_counts[ngram] = max(
max_ref_counts[ngram], ref_ngrams[ngram]
)
clipped = sum(
min(count, max_ref_counts[ngram])
for ngram, count in candidate_ngrams.items()
)
total = sum(candidate_ngrams.values())
return clipped / total if total > 0 else 0.0
def sentence_bleu(candidate, references, max_n=4):
candidate_len = len(candidate.split())
ref_lens = [len(r.split()) for r in references]
closest_ref_len = min(ref_lens, key=lambda l: abs(l - candidate_len))
bp = (
1.0
if candidate_len >= closest_ref_len
else math.exp(1 - closest_ref_len / candidate_len)
)
log_avg = 0.0
for n in range(1, max_n + 1):
p_n = bleu_precision(candidate, references, n)
log_avg += (1 / max_n) * math.log(p_n + 1e-10)
return bp * math.exp(log_avg)
all_captions = list(beam_captions.values()) + sampling_captions
bleu_scores = {}
for i, candidate in enumerate(all_captions):
references = [c for j, c in enumerate(all_captions) if j != i]
bleu_scores[i] = sentence_bleu(candidate, references)
caption_types = [f"Beam(n={n})" for n in beam_captions.keys()] + [
f"Sample{i + 1}" for i in range(len(sampling_captions))
]BLEU cross-consistency scores: --------------------------------------------- Beam(n=1) 1.000 ######################################## Beam(n=3) 1.000 ######################################## Beam(n=5) 1.000 ######################################## Sample1 0.001 Sample2 1.000 ######################################## Sample3 0.000 Higher BLEU = more similar to the other captions. Beam captions tend to be more consistent with each other.
VQA Answer Type Distribution
To understand the diversity of reasoning required in VQA benchmarks, let us visualize the distribution of answer types in a representative dataset. This distribution reveals both the composition of typical VQA benchmarks and the potential failure modes that arise from class imbalance.
# Representative proportions from the VQA v2 dataset structure
answer_type_distribution = {
"Yes/No": 38.4,
"Number": 12.3,
"Color": 8.9,
"Object": 14.2,
"Activity": 11.8,
"Location": 7.6,
"Other": 6.8,
}
The dominance of Yes/No questions changes the interpretation for model evaluation. A model that simply answers "yes" to every question would achieve roughly 38% accuracy without understanding any image. This is why balanced datasets like VQA v2 pair complementary images to make shortcut answers impossible. Even so, Yes/No questions remain harder to use as a true test of visual reasoning because the binary answer space can still be gamed by other statistical patterns.
Attention Visualization
One of the most interpretable aspects of modern VQA models is their attention maps. These maps show which image regions the model focuses on when generating each answer. Different question types naturally direct attention to different regions: counting questions concentrate on individual object locations, while attribute questions spread attention more broadly across the relevant region.
import numpy as np
def create_attention_map(h, w, peaks, sigma=0.12):
"""Create a Gaussian mixture attention map simulating spatial focus."""
y_grid = np.linspace(0, 1, h)
x_grid = np.linspace(0, 1, w)
xx, yy = np.meshgrid(x_grid, y_grid)
attention = np.zeros((h, w))
for px, py, weight in peaks:
gauss = np.exp(-((xx - px) ** 2 + (yy - py) ** 2) / (2 * sigma**2))
attention += weight * gauss
attention = attention / attention.max()
return attention
np.random.seed(42)
# "How many cats?" -- concentrated at individual cat locations
attention_counting = create_attention_map(
32, 32, peaks=[(0.25, 0.55, 0.7), (0.72, 0.45, 0.6)], sigma=0.12
)
# "What color is the couch?" -- broader, spread across the couch region
attention_color = create_attention_map(
32,
32,
peaks=[(0.25, 0.55, 0.5), (0.72, 0.45, 0.4), (0.5, 0.3, 0.3)],
sigma=0.18,
)
The counting question focuses on individual object locations, while the color question distributes attention more broadly across the couch region. This is the cross-modal attention mechanism at work: the question text steers which parts of the visual representation get emphasized when creating the answer. In real transformer-based VQA models, multiple attention heads perform this kind of selection in parallel, with different heads potentially specializing in different aspects of the visual scene.
Metric Comparison Across Captioning Approaches
Different evaluation metrics capture different aspects of caption quality. BLEU measures exact n-gram overlap, which penalizes valid paraphrases. CIDEr rewards content-specific vocabulary. Understanding how these metrics behave on the same captions reveals why no single metric is a perfect proxy for caption quality.
# Human reference captions for the COCO image (two cats on a couch)
human_references = [
"two cats laying on a pink blanket",
"a couple of cats are resting on a pink couch",
"two cats sleeping on top of a pink blanket",
"two cats lying on a pink couch",
"a pair of cats resting on a pink sofa",
]
def compute_cider_like(candidate, references):
"""Compute a simplified CIDEr-like score using TF-IDF-weighted n-gram overlap."""
all_texts = references + [candidate]
doc_count = len(all_texts)
# Compute TF-IDF weights for 1-grams and 2-grams
ngram_doc_freq = Counter()
for text in all_texts:
ngrams_in_doc = set()
for n in [1, 2]:
for ng in compute_ngrams(text, n):
ngrams_in_doc.add(ng)
for ng in ngrams_in_doc:
ngram_doc_freq[ng] += 1
def get_tfidf_vector(text):
vec = {}
for n in [1, 2]:
for ng, cnt in compute_ngrams(text, n).items():
idf = math.log((doc_count + 1) / (ngram_doc_freq[ng] + 1))
vec[ng] = cnt * idf
return vec
cand_vec = get_tfidf_vector(candidate)
scores = []
for ref in references:
ref_vec = get_tfidf_vector(ref)
# Cosine similarity between TF-IDF-weighted n-gram vectors
dot = sum(cand_vec.get(ng, 0) * ref_vec.get(ng, 0) for ng in ref_vec)
cand_norm = math.sqrt(sum(v**2 for v in cand_vec.values()) + 1e-10)
ref_norm = math.sqrt(sum(v**2 for v in ref_vec.values()) + 1e-10)
scores.append(dot / (cand_norm * ref_norm))
return sum(scores) / len(scores)
# Evaluate beam=5 caption and all sample captions
eval_captions = {
"Beam (n=5)": beam_captions[5],
"Sample 1": sampling_captions[0],
"Sample 2": sampling_captions[1],
"Sample 3": sampling_captions[2],
}
metric_results = {}
for label, caption in eval_captions.items():
bleu1 = bleu_precision(caption, human_references, 1)
bleu4 = sentence_bleu(caption, human_references, max_n=4)
cider_like = compute_cider_like(caption, human_references)
metric_results[label] = {
"BLEU-1": bleu1,
"BLEU-4": bleu4,
"CIDEr-like": cider_like,
}Metric comparison against human reference captions: Caption BLEU-1 BLEU-4 CIDEr-like ---------------------------------------------- Beam (n=5) 1.000 0.512 0.112 Sample 1 1.000 0.001 0.104 Sample 2 1.000 0.512 0.112 Sample 3 0.429 0.001 0.048
The metric comparison reveals an important tension in captioning evaluation: BLEU-4 is very sensitive to exact phrase matches, so short or differently-worded captions score poorly even when they accurately describe the image. CIDEr-like scores reward captions that use content-specific vocabulary shared with multiple references, making the metric more reliable to paraphrases that use the same specific words. Neither metric perfectly captures whether a caption is accurate and informative; they each capture a different facet of textual similarity.