Perceptual and Generative Evaluation › Perceptual Quality and Distributional Metrics

Human observers are not pixel sensors. Two images that differ in MSE by a factor of four can look nearly identical, and two that agree numerically can look very different. Learned perceptual metrics were introduced to close that gap, and distributional metrics were introduced to handle the fact that a stochastic future has no single correct answer. The next two subsections define each family precisely and state the assumptions under which the scores are meaningful.

Perceptual distances such as LPIPS still compare a prediction with a reference, using a feature comparison calibrated against human judgments. They do not remove stochastic reference ambiguity. FID and FVD compare collections through feature moments rather than aligned pairs; that changes the evaluation unit but can discard conditioning information. Neither change solves every reference problem.

LPIPS, from Zhang et al. (2018), compares two images using pretrained convolutional features rather than pixel coordinates. The recipe is:

• Feed both images through a pretrained backbone (typically AlexNet or VGG on ImageNet).
• Extract feature maps at several layers.
• Normalize the channel vector at each spatial location to unit $L_2$ norm, with the implementation's numerical stabilization.
• Take the squared difference of the normalized features at each layer.
• Apply learned channel weights to the differences, sum over channels, and average over spatial positions.
• Sum the layer contributions. The calibration weights are fitted to human perceptual judgments on a specific dataset.

The intuition behind the recipe is that early layers in a supervised network respond to edges and textures, while later layers respond to semantic content. Comparing several layers can therefore use different kinds of visual information, but the calibrated distance is not a general guarantee of semantic correctness. Unit-normalizing the channel vector removes its overall scale, apart from numerical stabilization; it does not balance individual channels, and relatively large channels can still dominate its direction. The per-channel coefficients are fitted so that the combined distance better agrees with the calibration judgments. Those coefficients do not necessarily identify which individual features people notice or what causally drives their verdicts.

In the original formulation, the squared difference is taken on the unit-normalized feature map $\hat\phi^{(l)}$. For a raw feature map $\phi^{(l)}(x) \in \mathbb{R}^{C_l \times H_l \times W_l}$ at layer $l$, the LPIPS distance between two images $x$ and $y$ is defined as:

$$\mathrm{LPIPS}(x,y) = \sum_l \frac{1}{H_l W_l} \sum_{i,j}\sum_c w_{lc}^{\,2}\Bigl(\hat\phi^{(l)}_{cij}(x)-\hat\phi^{(l)}_{cij}(y)\Bigr)^2$$

where:

• $x, y$: the two input images being compared
• $\phi^{(l)}(x)$: the feature map at layer $l$ of the pretrained backbone applied to $x$, of shape $C_l \times H_l \times W_l$
• $\hat\phi^{(l)}_{cij}(x)$: the channel-normalized activation at channel $c$, spatial position $(i,j)$, layer $l$ for image $x$
• $C_l, H_l, W_l$: number of channels, height, and width of the feature map at layer $l$
• $w_{lc}$: the channel calibration weight for channel $c$ at layer $l$ in the paper's squared weighted-norm notation. The learned coefficient on a squared channel difference is $w_{lc}^{2}$.

The expression is a sum of squared, weighted feature differences, not an ordinary Euclidean distance with a guaranteed triangle inequality. Similar normalized feature maps give a small score even when raw pixels differ. Classification training and channel normalization do not guarantee invariance to input brightness or color changes. Its perceptual interpretation comes from the reported human calibration and evaluation, not a universal invariance theorem.

This formula describes the original calibrated comparison with a fixed feature backbone. Calibration learns channel coefficients; the backbone already contains parameters learned during its own training. Report the metric implementation and version rather than assuming all feature-distance variants use this exact convention.

It is easy to misread LPIPS as "a universal perceptual distance." It is not. Three sources of dependence matter:

• The backbone. AlexNet and VGG can produce different numbers on the same pair. Scores can also differ between ImageNet-trained and out-of-domain backbones.
• The preprocessing. Image resizing, normalization statistics, and channel order (RGB vs BGR) shift the numbers.
• The training set of the linear weights. The weights are calibrated against human judgments on the specific datasets (e.g., BAPPS) used in the original paper. On distributions far from that calibration, the metric is being extrapolated.

LPIPS is a full-reference perceptual-similarity proxy, not a no-reference realism detector. A beautifully rendered but physically impossible future is not guaranteed a low LPIPS: the score also depends on its reference. Conversely, a low score does not certify physical correctness. Perceptual similarity and correct physics are separate questions.

LPIPS does not measure realism without a reference. An image compared with itself has zero feature difference whether it shows noise or a detailed scene. Two independently drawn images from the same output style need not have near-zero LPIPS. In stochastic prediction, a valid alternative future can also differ perceptually from the particular realized reference.

FID, introduced by Heusel et al. (2017), compares two image sample populations without requiring paired reference images. It is the squared 2-Wasserstein distance between multivariate Gaussians fitted to their Inception feature distributions. Let $\mu_r, \Sigma_r$ denote real-image feature moments and $\mu_g, \Sigma_g$ generated-image feature moments. The Gaussian distance is:

$$\mathrm{FID} \;=\; \lVert \mu_r - \mu_g \rVert_2^2 \;+\; \mathrm{Tr}\!\left( \Sigma_r + \Sigma_g - 2\bigl(\Sigma_r^{1/2} \Sigma_g \Sigma_r^{1/2}\bigr)^{1/2} \right)$$

where:

• $\mu_r, \Sigma_r$: the mean and covariance of features of real images under the Inception network
• $\mu_g, \Sigma_g$: the mean and covariance of features of generated images under the same encoder
• $\lVert \mu_r - \mu_g \rVert_2^2$: the squared Euclidean distance between the two feature means, capturing a shift in average feature location
• $\mathrm{Tr}(\cdot)$: the matrix trace, summing the diagonal entries
• $\Sigma_r^{1/2}$: the symmetric positive-semidefinite square root of $\Sigma_r$
• $\bigl(\Sigma_r^{1/2} \Sigma_g \Sigma_r^{1/2}\bigr)^{1/2}$: the matrix square root appearing in the Bures distance between the two covariances, capturing a difference in feature spread and correlation

The trace term is the squared Bures distance on positive-semidefinite covariance matrices. The symmetric matrix inside the inner square root is positive-semidefinite, so its positive-semidefinite square root is well-defined. The full expression is the squared 2-Wasserstein distance between the two fitted Gaussians under a fixed encoder, and is zero if and only if those Gaussians have the same mean and covariance.

The first term measures a shift in the center of the feature cloud. The second term measures a mismatch in the shape of the cloud, including spread and correlation between feature dimensions. Both are computed on features, not on pixels, so FID never asks whether a specific generated frame looks like a specific real frame. It asks whether the statistics of the two feature clouds match.

FID is not a distance between the full distributions. It compares first and second moments of Inception features. Differences in higher moments, in tail behavior, or in the actual support of either distribution are invisible to FID. Two clearly different image distributions can have identical FID of zero if their Inception features share the same mean and covariance.

The moment-blindness claim is visible in one dimension. A single Gaussian and a two-mode mixture can share a mean and variance while looking completely different.

FID uses Gaussian fits to feature means and covariances, not feature histograms or pixel-aligned reference pairs.

The phrase "identical FID of zero" is not a hypothetical edge case. Any two distributions with the same first two moments of Inception features score identically under FID, by construction. If one of them happens to place all its mass on a small set of atypical images while the other spreads mass broadly, FID still returns zero. Higher-order structure, all the way down to which specific images appear, is unconstrained.

Three practical issues come with FID.

• Finite-sample estimation. In practice the moments are estimated from finite samples: $\hat\mu_r, \hat\Sigma_r$ from real features and $\hat\mu_g, \hat\Sigma_g$ from generated features. A centered covariance estimate from $n$ vectors in $d$ dimensions has rank at most $\min(d,n-1)$, so it is necessarily rank-deficient when $n\le d$. Chong and Forsyth (CVPR 2020), in "Effectively Unbiased FID and Inception Score and Where to Find Them", show finite-sample, model-dependent FID bias. Rank deficiency alone does not imply a universally downward bias, and using the same sample count does not eliminate differences in estimator bias between models. Report real and generated sample counts and the estimation protocol.
• Encoder dependence. Conventional FID uses Inception-v3 features pretrained for ImageNet classification, not a feature distance fitted directly to human judgments. Domain changes can alter which distinctions the features retain. Validate their relevance before interpreting scores on a new image domain.
• Preprocessing and protocol sensitivity. Image resizing, interpolation, JPEG compression, and whether the model produces the same number of samples as the reference all affect FID. The right discipline is to hold these constant when comparing models and to report them.

A small FID difference may be comparable to estimation noise or model-dependent bias. A bootstrap interval can describe sampling uncertainty, but it does not automatically remove that bias. Resample independent units: clips from the same trajectory may require trajectory-level grouping. Repeated high-dimensional covariance square roots can also be costly. Report the sample counts, estimator, resampling unit, and interval construction.

FVD (Fréchet Video Distance), from Unterthiner et al. (2018), applies the Gaussian feature-distance construction to video clips, using spatiotemporal features such as I3D embeddings. State clip duration, frame rate, spatial resolution, encoder checkpoint, and feature layer. Changes in these can change the score. FVD can detect some temporal failures, but matching its feature moments does not certify object persistence, identity, or causality.

The practical implication is that perceptual and distributional metrics are useful complements to paired-error metrics, not replacements. For either FID or FVD, state the encoder, sample counts, preprocessing, and spatial resolution. For FVD, also state clip duration and frame rate; for FID computed on video frames, state how frames were sampled. Still-image FID has no intrinsic clip-length parameter. When you report LPIPS, state the backbone and the weights. When you report PSNR, state the dynamic range. A single number without its protocol is not a measurement.

A pooled marginal can hide action-conditioning errors. Suppose two actions occur with equal weight and a predictor swaps their distinct future distributions. The pooled mixture is unchanged, so a metric of that marginal cannot detect the swap. A conditional comparison can reveal it if its features and statistic distinguish the two conditionals. Equal pooling weights matter; mirror-image shapes alone do not guarantee equality with unequal action frequencies.

Pooling discards the association between a condition and its output. Stratifying by action or history is one remedy; testing the joint condition-output distribution or using a proper conditional probabilistic score are others. A stratified moment metric still has encoder and moment-matching blind spots.

The next section demonstrates this pooling failure with a one-dimensional toy Gaussian-feature distance, not a published metric. The same loss of conditioning can affect pooled FID or FVD. Paired LPIPS and SSIM retain their specific references, so this argument does not apply to them in the same way.