Multimodal Evaluation › Multimodal Generation Evaluation

Evaluating generation presents fundamentally different challenges from understanding tasks. When a model generates an image from text (text-to-image) or describes an image (image captioning), we lack ground truth references in the same way we do for classification tasks.

The basic difference lies in the many-to-many mapping between modalities. For image captioning, multiple valid descriptions exist for any single image. For text-to-image generation, multiple pixel configurations might satisfy the same text prompt. This ambiguity makes evaluation inherently probabilistic rather than deterministic, requiring metrics that capture distributional similarity or semantic alignment rather than exact matches.

This problem is not unique to multimodal generation, as we encounter it in text generation evaluation as well (discussed in the chapter on machine translation evaluation). But multimodal generation adds extra dimensions of challenge: captions need linguistic quality and visual accuracy, while image generation requires semantic adherence to a text prompt and perceptual quality along dimensions like sharpness and coherence alongside realism.

Image captioning requires generating natural language descriptions of images. Evaluation metrics compare generated captions against human-written references, using statistical properties of the comparison to estimate quality.

BLEU (Bilingual Evaluation Understudy) and ROUGE were originally designed for machine translation and text summarization respectively. BLEU measures n-gram precision: the fraction of n-grams in the generated caption that appear in the reference captions. BLEU-4 specifically measures 4-gram overlap, and includes a brevity penalty to prevent very short captions from achieving artificially high precision by only using words that appear in references. The brevity penalty multiplies the n-gram precision by a factor less than 1 when the candidate is shorter than the shortest reference, penalizing the strategy of generating very short but highly precise captions.

However, n-gram metrics suffer from well-documented limitations in multimodal contexts. They penalize valid paraphrases: a model that generates "The feline is resting on the furniture" receives low BLEU scores against references saying "The cat is sitting on the couch," despite conveying the same information. They are also insensitive to word order in limited ways and entirely insensitive to semantic content beyond n-gram overlap. A nonsensical sentence that happens to share many words with a reference might outscore a coherent sentence that uses synonyms. This limitation motivated the development of semantically aware metrics.

METEOR (Metric for Evaluation of Translation with Explicit ORdering) addresses BLEU's synonym blindness by letting flexible matching through WordNet synonymy and stemming. METEOR recognizes that "running" and "jogging" describe the same action, and that "walked" and "walking" are morphological variants of the same concept. This linguistic awareness makes METEOR more tolerant of valid word choices while still penalizing semantically incorrect captions. METEOR also considers word order through a fragmentation penalty, but does so more flexibly than BLEU's strict n-gram counting.

CIDEr (Consensus-based Image Description Evaluation) takes a different approach by weighting n-grams according to their specificity. Like TF-IDF in document retrieval, CIDEr gives higher weight to rare, specific n-grams that appear in both the candidate and references. Common words like "the," "a," and "is" receive low weight because they appear in nearly all captions and carry little information. But specific phrases like "yellow fire hydrant" receive high weight because they stand for precise, informative content. This weighting makes CIDEr more sensitive to content specificity than BLEU, rewarding models that capture distinctive image details rather than just using common words. CIDEr is computed using TF-IDF vectors over n-grams from both the candidate and multiple references, then measuring cosine similarity between these vectors.

SPICE (Semantic Propositional Image Caption Evaluation) takes the most radical departure from n-gram overlap by parsing captions into scene graphs. Rather than comparing word sequences, SPICE converts captions to structured representations of objects and attributes plus relationships (for example, parsing "a large black dog runs in the grass" into objects: {dog, grass}, attributes: {large, black}, relationships: {runs-in}). It then computes F-score based on the overlap between candidate and reference scene graphs. SPICE captures whether models mentioned correct objects and relationships regardless of word order or surface form, which makes it reliable to valid paraphrases that would confuse n-gram metrics. However, SPICE depends on the quality of the scene graph parser, and parser errors can penalize correct captions that use unusual phrasings.

CLIPScore is a change from reference-based to reference-free evaluation. Rather than comparing a generated caption against human references, CLIPScore uses a contrastive vision-language model (specifically CLIP, discussed in the CLIP chapter) to measure how well a generated caption aligns with its source image. The caption and image are encoded into a shared embedding space, and their cosine similarity is the score. This approach requires no reference captions. This makes possible evaluation on entirely novel images where collecting references would be prohibitively expensive. CLIPScore correlates well with human judgment of caption quality and has become standard for evaluating image captioning models during development, where the speed advantage of reference-free evaluation lets rapid iteration.

The trade-offs between these metrics are real and matter for practical use. BLEU is fast and simple but severely penalizes synonyms. METEOR is more linguistically aware but depends on WordNet coverage, which is incomplete for domain-specific or informal language. CIDEr rewards specificity but is sensitive to corpus statistics and must be computed relative to a reference corpus. SPICE captures semantic structure but parser errors cascade into evaluation errors. CLIPScore requires no references but inherits CLIP's training biases. No single metric is universally superior; practitioners typically report multiple metrics to give a more complete picture of caption quality.

When models like DALL-E or Stable Diffusion generate images from text prompts, evaluation must assess both fidelity (does the image match the text?) and quality (is the image visually coherent?). These two dimensions are separable: a blurry, artifact-ridden image might accurately depict the described scene, while a photorealistic image might depict the wrong content entirely.

FID (Fréchet Inception Distance) measures the statistical distance between distributions of generated and real images in a feature space. The procedure works as follows: first, pass both real and generated images through a pre-trained Inception network to extract feature vectors. Second, compute the mean $\mu$ and covariance $\Sigma$ of these feature vectors for each set. Third, compute the Fréchet distance between the two resulting Gaussian distributions:

$$\text{FID} = \|\mu_X - \mu_Y\|^2 + \text{Tr}\!\left(\Sigma_X + \Sigma_Y - 2\!\sqrt{\Sigma_X \Sigma_Y}\right)$$

where:

• $\mu_X, \mu_Y$: mean feature vectors of the real and generated image distributions
• $\Sigma_X, \Sigma_Y$: covariance matrices of the real and generated feature distributions
• $\text{Tr}(\cdot)$: trace of a matrix (sum of diagonal elements)
• $\sqrt{\Sigma_X \Sigma_Y}$: matrix square root of the product of covariances, accounting for differences in how the distributions spread and correlate

Lower FID indicates that the generated distribution is statistically closer to the real distribution, suggesting higher quality and diversity. The metric captures both quality (low FID requires each generated image to look realistic) and diversity (FID penalizes mode collapse, where a generator produces many similar images, because the collapsed distribution would have very different covariance from the real distribution).

FID correlates reasonably well with human perception of visual quality, but it has necessary limitations. Most importantly, FID says nothing about text alignment. Two completely different images (one of a cat, one of a dog) might both produce good FID scores if they look realistic, regardless of whether the prompt asked for "a cat" or "a dog." A system that generates beautiful but topically irrelevant images would score well on FID. This limitation motivated the development of complementary alignment metrics.

IS (Inception Score) is an older metric that measures two properties simultaneously: diversity (the generated images should cover many distinct categories) and "objectness" (each individual image should clearly belong to some recognizable category). IS computes the KL divergence between the conditional class distribution $p(y \mid x)$ (what category does this specific image belong to?) and the marginal class distribution $p(y)$ (what categories appear across all generated images?). High IS requires that each image has low entropy in its class prediction (clear, recognizable objects) while the marginal distribution has high entropy (many different types of objects appear across the full set of images). Despite its intuitive appeal, IS has fallen out of favor because it is sensitive to the Inception model's classification categories and fails to detect certain failure modes like mode collapse within a category.

CLIP Score has become the standard metric for evaluating text-to-image alignment. It measures cosine similarity between the text prompt embedding and the generated image embedding in CLIP's shared representation space. A high CLIP score means the image content resembles what CLIP's vision encoder would predict from seeing that text, showing semantic alignment between prompt and image. This is directly analogous to CLIPScore for captioning, but applied in the reverse direction: rather than evaluating whether a caption matches an image, we evaluate whether an image matches a caption.

CLIP Score can be fooled in predictable ways. Images that contain the right conceptual elements but in wrong spatial configurations (a "dog on the left of the cat" where they are reversed) may still score highly because CLIP encodes overall semantic content rather than precise spatial arrangements. Images with watermarks or overlaid text matching the prompt might also score highly despite visual quality issues. These edge cases motivate complementary use of human evaluation.

Human Evaluation remains the gold standard for text-to-image generation. Human evaluators rate images on multiple dimensions:

• Image-text alignment: Does the image accurately depict the prompt, including all specified attributes and relationships?
• Image quality: Is the image visually coherent, free of artifacts, and perceptually realistic or artistically intentional?
• Aesthetic quality: Does the image exhibit compositional balance, appropriate use of color, and visual appeal according to human taste?
• Safety and appropriateness: Does the image avoid harmful, biased, or offensive content?

Human evaluation captures nuances that automated metrics miss, including artistic style, cultural appropriateness, and fine-grained compositional details like "a person on the left side of a park bench" versus "a person on the right side." However, human evaluation is expensive (each comparison costs fractions of a dollar but aggregate costs add up quickly at scale), slow (human studies take days to complete compared to milliseconds for automated metrics), and subject to individual variation in taste, cultural background, and interpretation. The standard practice for publication is to conduct human evaluation studies using methodological controls like inter-annotator agreement measures and randomized presentation order to mitigate these sources of noise.

Extending evaluation to video introduces temporal consistency requirements that don't arise for static images. Video generation must produce visually realistic individual frames and coherent sequences where objects maintain their appearance, move according to physical laws, and respond to causes specified in the prompt.

FVD (Fréchet Video Distance) extends FID to the temporal domain by using video understanding models (typically 3D convolutional networks or video transformers) to extract features that capture both spatial appearance and temporal dynamics. The same distributional distance computation as FID applies, but the features now encode how content changes over time, not just what content appears in a single frame. Lower FVD indicates that the generated videos are more statistically similar to real videos in terms of both content and motion patterns.

Frame consistency metrics use optical flow to verify temporal coherence. Optical flow measures how pixel values move between consecutive frames; unrealistic flow patterns (sudden jumps, flickering, or objects that teleport) indicate temporal incoherence. Consistency scores measure the variance of appearance features across frames (after accounting for legitimate motion), penalizing models that generate frames independently without maintaining object identity over time.

Text-video alignment extends CLIP-style metrics to temporal content by averaging alignment scores across sampled frames or using video-language models that encode full temporal sequences. A video prompt "a red ball rolling down a hill" should produce frames containing a red ball and a hill, plus progressive motion of the ball down the slope, which requires both semantic alignment and temporal coherence.

Video generation adds the dimension of temporal causality: objects must appear correct in individual frames and move according to physical laws and the prompt's specifications. This makes video generation evaluation substantially more complex than image generation evaluation, and the field is still developing reliable metrics that capture temporal quality as reliably as FID captures static image quality.