Image Generation › Evaluation of Generated Images

Evaluating image generation quality is difficult. Human perception is the gold standard, but human evaluation is expensive and slow, with limited reproducibility. Several automated metrics have been developed, each capturing different aspects of quality.

FID is the most widely used automated metric for measuring image generation quality. It measures the distance between the distribution of generated images and the distribution of real images, not in pixel space, but in the feature space of a pre-trained InceptionV3 network.

The idea is that features from a network trained on ImageNet capture semantically meaningful aspects of image content: textures and shapes alongside structures and high-level objects. If the distribution of these features in the generated images matches the distribution in the real images, the generated images capture the same semantic variety and realism as the real data.

Given real images $\mathbf{X}_r$ and generated images $\mathbf{X}_g$, their Inception features are extracted and fit to multivariate Gaussians with means $\boldsymbol{\mu}_r$, $\boldsymbol{\mu}_g$ and covariances $\boldsymbol{\Sigma}_r$, $\boldsymbol{\Sigma}_g$. FID computes the Frechet distance between these Gaussians:

$$\text{FID} = \|\boldsymbol{\mu}_r - \boldsymbol{\mu}_g\|^2 + \text{Tr}\left(\boldsymbol{\Sigma}_r + \boldsymbol{\Sigma}_g - 2(\boldsymbol{\Sigma}_r \boldsymbol{\Sigma}_g)^{1/2}\right)$$

where:

• $\|\boldsymbol{\mu}_r - \boldsymbol{\mu}_g\|^2$: squared distance between mean feature vectors, measuring distributional shift in average content
• $\text{Tr}(\cdot)$: trace of a matrix (sum of diagonal elements)
• $(\boldsymbol{\Sigma}_r \boldsymbol{\Sigma}_g)^{1/2}$: matrix square root of the product of covariances, measuring structural similarity

Lower FID is better: a perfect model matching the real data distribution exactly would achieve FID of 0. State-of-the-art models achieve FID scores below 3 on standard benchmarks like MS-COCO or ImageNet.

FID has known limitations. It is sensitive to the number of samples used: you need at least 10,000 generated images for reliable estimation, and the value changes substantially with fewer samples. It depends on the specific feature extractor (InceptionV3 may not capture the aspects of quality most relevant to human preference). It measures the full distribution, so a model that generates many diverse but lower-quality images can outscore one that generates fewer but higher-quality images on FID. Despite these limitations, it remains the standard comparison metric in published research.

For text-to-image evaluation, CLIP Score measures the semantic alignment between a generated image and its text prompt, directly using the CLIP model's joint embedding space.

CLIP Score computes the cosine similarity between the CLIP image embedding of the generated image and the CLIP text embedding of the prompt:

$$\text{CLIP Score} = w \cdot \max(0, \cos(\mathbf{e}_{\text{image}}, \mathbf{e}_{\text{text}}))$$

where:

• $\mathbf{e}_{\text{image}}$: the normalized CLIP image embedding of the generated image
• $\mathbf{e}_{\text{text}}$: the normalized CLIP text embedding of the prompt
• $w$: a scaling factor (typically 100) that maps cosine similarities into a human-interpretable score range
• $\max(0, \cdot)$: clips negative similarities to zero, since negative cosine similarity indicates opposite directions in the embedding space

A score of 25-30 is typical for high-quality text-image aligned models. CLIP Score captures semantic fidelity well: does the image visually depict the concepts mentioned in the text? But it does not capture visual quality or photorealism, and it has blind spots wherever CLIP's training data was sparse. It is also susceptible to adversarial inputs: images that score highly on CLIP but look strange to human observers. It is most useful for comparing text-image alignment across models or studying how different prompting strategies affect faithfulness, not as a standalone quality measure.

Automated metrics often disagree with human judgment about what makes an image "good." Human evaluation remains the gold standard for capturing subjective quality dimensions that metrics miss.

Human evaluation typically asks annotators to:

• Compare two images generated from the same prompt and indicate which is better (pairwise preference comparison)
• Rate an image on overall quality and photorealism, text-image alignment, diversity, aesthetic appeal
• Evaluate specific failure modes: distorted hands, unnatural textures, implausible lighting, artifacts in background regions, or incorrect object counts

Pairwise comparisons are often more reliable than absolute ratings because they avoid systematic biases in how different annotators interpret rating scales. The Bradley-Terry model can convert pairwise preferences into global scores. Modern leaderboards like HPSv2 and ELO-based systems collect large numbers of human pairwise comparisons and use them to rank models.

Human evaluation is expensive to scale to thousands of prompts and model comparisons, but it remains the most reliable signal for the aspects of quality that matter most in practice: does the image look good to a human? Would a person be satisfied receiving this image in response to their prompt?

Precision and recall for generative models, as defined by Kynkaanniemi et al. (2019), measure two distinct aspects of generation quality:

• Precision: Do generated images look realistic? Are they within the manifold of real images? High precision means the model rarely generates implausible images that fall outside what real images look like.
• Recall: Does the model cover the full diversity of the real distribution? High recall means the model can generate images across the full range of visual variety present in the training data.

A model that generates only highly photorealistic images of a narrow slice of the distribution would have high precision but low recall: it generates quality images but misses most of what real images look like. A model that generates highly varied images that are sometimes blurry or contain artifacts would have high recall but low precision.

These metrics are computed using nearest-neighbor distances in the InceptionV3 feature space. For precision, we ask: for each generated image, does a real image exist nearby? For recall: for each real image, does a generated image exist nearby?

Precision and recall provide more detail than FID alone. FID is a single scalar that conflates fidelity and diversity; precision and recall decompose these into separate measurements, allowing diagnosis of whether a model's failure mode is generating unrealistic images (low precision) or missing diversity (low recall). Increasing guidance scale, for example, typically increases precision (higher quality) at the cost of recall (less diversity).