Perceptual and Generative Evaluation

An action can have more than one plausible future. In the small example below, a blob moves either a short or a long distance to the right. The realized future takes the short step. A sharp prediction of the long step puts the blob in the wrong place; averaging the two possible frames produces a faded, two-blob image with lower squared pixel error than that wrong sharp prediction. Yet the sharp prediction matching the realized short step has zero error and beats both. Which comparison should an evaluation report?

The example separates three questions: does an output look plausible, does it match a particular realized future, and does the predictive distribution match the possible futures and their probabilities? These properties can disagree. A score may provide evidence about more than one property, but no single score discussed here certifies all three. This chapter explains what each score measures and what remains unchecked.

We will compute the pixel scores, but we will not run a human-preference study or a learned perceptual network on this toy. Its faded average illustrates why an expected-error optimum need not resemble a sample. Whether people prefer either output requires a separate experiment. The useful task is to choose scores whose questions match the intended use.

This chapter sits inside Part XI: Evaluation and Understanding, after The World-Model Evaluation Ladder. That ladder distinguishes one-step prediction, fixed-action open-loop rollouts, interventions, closed-loop decisions, and transfer. Here we focus on scoring observations under declared prediction protocols. Observation scores can reveal useful failures, but they cannot by themselves certify state, physics, or causal fidelity. The next chapter develops those diagnostics.

The scope is observation evaluation, not a certificate of useful internal state or decision quality. Pixels, feature distances, likelihoods, and human judgments provide different kinds of evidence. A table of those scores becomes interpretable only when the evaluation protocol is stated.

We begin with a synthetic scene and score hand-designed predictions against its realized future. The example requires neither checkpoints nor network training. We then distinguish the questions answered by MSE, PSNR, and SSIM from those answered by LPIPS, FID, and FVD. For stochastic futures, we examine conditional scoring, future-reference best-of-$K$ selection, and an action reversal that pooling conceals. Human judgment protocols and learned evaluators complete the scorecard.

A 48 by 48 grayscale canvas lets us inspect the pixels and recompute the scores. Its two equally probable futures illustrate the difference between matching one realization and describing a distribution. The example does not reproduce the architecture, data, or failure modes of a large video model.

As in Chapter 61, keep three prediction regimes separate in your head:

• Teacher-forced one-step prediction: the model receives the realized observation or history and the action needed to predict the next observation. Errors from its earlier predictions are not fed back as context.
• Fixed-action open-loop rollout: the model receives $h_t$ and actions $a_{t:t+H-1}$ and predicts $o_{t+1:t+H}$ without seeing intermediate realized observations. Earlier prediction errors can affect later predictions, although error need not increase monotonically.
• Closed-loop decision: an agent uses the model inside a planner or policy and receives environment feedback. Task return evaluates decision usefulness; image scores remain diagnostic. Planning, Control, and Policy Evaluation covers that regime.

An MSE of 0.003 needs context: which regime produced it, how many steps were rolled out, and what was averaged? Changing the regime can change the score substantially even with the same model. Keep the regimes separate when reading a prediction table.

The following curves are prescribed illustrations, not measurements from a trained predictor. We construct a gently increasing teacher-forced curve and an exponentially increasing open-loop curve to illustrate a possible difference between protocols. The teacher-forced curve is not constant, and neither shape is a universal law.

Everything we build below uses the convention $p_\theta(o_{t+1:t+H} \mid h_t, a_{t:t+H-1})$, where:

• $o_{t+1:t+H}$: the sequence of future observations the model predicts, from time $t+1$ through time $t+H$
• $h_t = (o_{\le t}, a_{<t})$: the history, consisting of observations up to and including time $t$ and actions strictly before time $t$
• $a_{t:t+H-1}$: the sequence of actions from time $t$ to time $t+H-1$, one action per predicted transition
• $H$: the prediction horizon, i.e. the number of future frames or steps predicted
• $\theta$: the model parameters

History can supply predictive information missing from a single frame in partially observed settings. It is not, by itself, a definition of a world model or a guarantee that the history is sufficient. Supplying actions lets us ask conditional prediction questions; observational action conditioning alone does not identify the effects of causal interventions.

Use $H_{\mathrm{img}}$ for image height and reserve $H$ for the prediction horizon. Grayscale frames have shape $(H_{\mathrm{img}},W)$ and color frames $(H_{\mathrm{img}},W,C)$, with values in $[0,1]$. Here $W$ is width and $C$ is the channel count. Other layouts, such as channels-first, change the axes to average but not the metric's meaning. A rollout adds a time dimension whose length is $H$.

Reconstruction scoring and future-prediction scoring agree on the math and disagree on the story. In reconstruction the model sees $o_t$, produces $\hat o_t$, and we compare $\hat o_t$ to $o_t$. In future prediction the model sees $h_t$ and $a_{t:t+H-1}$ and produces $\hat o_{t+1:t+H}$, and we compare the predicted sequence to the realized sequence $o_{t+1:t+H}$. Pixel-wise mean squared error (MSE) and peak signal-to-noise ratio (PSNR) are the two workhorses of both. Frame MSE averages squared pixel residuals; frame PSNR applies a logarithm to that spatially averaged MSE. For sequences, averaging frame PSNR is not the same as applying the logarithm to a pooled MSE. The averaging conventions matter and are the subject of the next formulas.

The reference changes meaning between tasks. Ordinary reconstruction compares against the fixed input observation; zero pixel MSE requires reproducing that array, but reconstruction quality alone does not establish useful internal state. Future prediction compares against a realized outcome, potentially one of several possible futures. The same distance therefore has a different interpretation.

The mean squared error (MSE) between a prediction $\hat x$ and a target $x$, both of shape $(H_{\mathrm{img}}, W)$ with pixel values in $[0,1]$, is

where:

• $\hat x_{ij}$: the predicted pixel value at spatial position $(i, j)$
• $x_{ij}$: the target pixel value at the same position
• $H_{\mathrm{img}}, W$: the height and width of the frame in pixels
• $\frac{1}{H_{\mathrm{img}}W}$: normalization by the pixel count, making the result an average rather than a sum

The subtraction $\hat x_{ij} - x_{ij}$ is the signed per-pixel error. Squaring removes its sign and heavily penalizes large deviations.

The square grows quadratically with error magnitude. Uniform error of 0.1 gives MSE 0.01. Error of 0.3 on 10 percent of pixels, with zero error elsewhere, gives MSE $0.09\times0.1=0.009$. These close averages do not describe where the errors occur. The perceptual importance of a localized error depends on the scene and viewing task; we have not measured it in this calculation.

The quadratic penalty is easiest to see as a curve. The same squared-error contribution can come from a small error everywhere or a larger error in a small region.

The bar chart makes the first-class concern of this chapter concrete. Pixel error rewards the model that got lucky on this particular draw, not the model that captured the conditional. Sharp-B is a mode the ground-truth process itself can produce; if the realized future had been Sharp-B instead, the ranking would flip. And the conditional mean, which is MSE-optimal on average, looks neither like Sharp-A nor Sharp-B in this particular draw.

One paired score evaluates one prediction against one realized future. Repeated paired MSE estimates expected squared error, but even that expectation does not characterize the full conditional distribution: the conditional mean is optimal without representing its modes or probabilities. Distributional tests with explicit conditioning or proper probabilistic scores answer a different question.

MSE remains useful for measuring paired accuracy. To assess the forecast distribution, use a declared probabilistic scoring rule or conditional distributional diagnostic over enough held-out cases. Merely averaging more paired MSE values does not turn MSE into a complete distributional test.

Human observers are not pixel sensors. Two images that differ in MSE by a factor of four can look nearly identical, and two that agree numerically can look very different. Learned perceptual metrics were introduced to close that gap, and distributional metrics were introduced to handle the fact that a stochastic future has no single correct answer. The next two subsections define each family precisely and state the assumptions under which the scores are meaningful.

Perceptual distances such as LPIPS still compare a prediction with a reference, using a feature comparison calibrated against human judgments. They do not remove stochastic reference ambiguity. FID and FVD compare collections through feature moments rather than aligned pairs; that changes the evaluation unit but can discard conditioning information. Neither change solves every reference problem.

LPIPS, from Zhang et al. (2018), compares two images using pretrained convolutional features rather than pixel coordinates. The recipe is:

• Feed both images through a pretrained backbone (typically AlexNet or VGG on ImageNet).
• Extract feature maps at several layers.
• Normalize the channel vector at each spatial location to unit $L_2$ norm, with the implementation's numerical stabilization.
• Take the squared difference of the normalized features at each layer.
• Apply learned channel weights to the differences, sum over channels, and average over spatial positions.
• Sum the layer contributions. The calibration weights are fitted to human perceptual judgments on a specific dataset.

The intuition behind the recipe is that early layers in a supervised network respond to edges and textures, while later layers respond to semantic content. Comparing several layers can therefore use different kinds of visual information, but the calibrated distance is not a general guarantee of semantic correctness. Unit-normalizing the channel vector removes its overall scale, apart from numerical stabilization; it does not balance individual channels, and relatively large channels can still dominate its direction. The per-channel coefficients are fitted so that the combined distance better agrees with the calibration judgments. Those coefficients do not necessarily identify which individual features people notice or what causally drives their verdicts.

In the original formulation, the squared difference is taken on the unit-normalized feature map $\hat\phi^{(l)}$. For a raw feature map $\phi^{(l)}(x) \in \mathbb{R}^{C_l \times H_l \times W_l}$ at layer $l$, the LPIPS distance between two images $x$ and $y$ is defined as:

where:

• $x, y$: the two input images being compared
• $\phi^{(l)}(x)$: the feature map at layer $l$ of the pretrained backbone applied to $x$, of shape $C_l \times H_l \times W_l$
• $\hat\phi^{(l)}_{cij}(x)$: the channel-normalized activation at channel $c$, spatial position $(i,j)$, layer $l$ for image $x$
• $C_l, H_l, W_l$: number of channels, height, and width of the feature map at layer $l$
• $w_{lc}$: the channel calibration weight for channel $c$ at layer $l$ in the paper's squared weighted-norm notation. The learned coefficient on a squared channel difference is $w_{lc}^{2}$.

The expression is a sum of squared, weighted feature differences, not an ordinary Euclidean distance with a guaranteed triangle inequality. Similar normalized feature maps give a small score even when raw pixels differ. Classification training and channel normalization do not guarantee invariance to input brightness or color changes. Its perceptual interpretation comes from the reported human calibration and evaluation, not a universal invariance theorem.

This formula describes the original calibrated comparison with a fixed feature backbone. Calibration learns channel coefficients; the backbone already contains parameters learned during its own training. Report the metric implementation and version rather than assuming all feature-distance variants use this exact convention.

It is easy to misread LPIPS as "a universal perceptual distance." It is not. Three sources of dependence matter:

• The backbone. AlexNet and VGG can produce different numbers on the same pair. Scores can also differ between ImageNet-trained and out-of-domain backbones.
• The preprocessing. Image resizing, normalization statistics, and channel order (RGB vs BGR) shift the numbers.
• The training set of the linear weights. The weights are calibrated against human judgments on the specific datasets (e.g., BAPPS) used in the original paper. On distributions far from that calibration, the metric is being extrapolated.

LPIPS is a full-reference perceptual-similarity proxy, not a no-reference realism detector. A beautifully rendered but physically impossible future is not guaranteed a low LPIPS: the score also depends on its reference. Conversely, a low score does not certify physical correctness. Perceptual similarity and correct physics are separate questions.

LPIPS does not measure realism without a reference. An image compared with itself has zero feature difference whether it shows noise or a detailed scene. Two independently drawn images from the same output style need not have near-zero LPIPS. In stochastic prediction, a valid alternative future can also differ perceptually from the particular realized reference.

FID, introduced by Heusel et al. (2017), compares two image sample populations without requiring paired reference images. It is the squared 2-Wasserstein distance between multivariate Gaussians fitted to their Inception feature distributions. Let $\mu_r, \Sigma_r$ denote real-image feature moments and $\mu_g, \Sigma_g$ generated-image feature moments. The Gaussian distance is:

where:

• $\mu_r, \Sigma_r$: the mean and covariance of features of real images under the Inception network
• $\mu_g, \Sigma_g$: the mean and covariance of features of generated images under the same encoder
• $\lVert \mu_r - \mu_g \rVert_2^2$: the squared Euclidean distance between the two feature means, capturing a shift in average feature location
• $\mathrm{Tr}(\cdot)$: the matrix trace, summing the diagonal entries
• $\Sigma_r^{1/2}$: the symmetric positive-semidefinite square root of $\Sigma_r$
• $\bigl(\Sigma_r^{1/2} \Sigma_g \Sigma_r^{1/2}\bigr)^{1/2}$: the matrix square root appearing in the Bures distance between the two covariances, capturing a difference in feature spread and correlation

The trace term is the squared Bures distance on positive-semidefinite covariance matrices. The symmetric matrix inside the inner square root is positive-semidefinite, so its positive-semidefinite square root is well-defined. The full expression is the squared 2-Wasserstein distance between the two fitted Gaussians under a fixed encoder, and is zero if and only if those Gaussians have the same mean and covariance.

The first term measures a shift in the center of the feature cloud. The second term measures a mismatch in the shape of the cloud, including spread and correlation between feature dimensions. Both are computed on features, not on pixels, so FID never asks whether a specific generated frame looks like a specific real frame. It asks whether the statistics of the two feature clouds match.

FID is not a distance between the full distributions. It compares first and second moments of Inception features. Differences in higher moments, in tail behavior, or in the actual support of either distribution are invisible to FID. Two clearly different image distributions can have identical FID of zero if their Inception features share the same mean and covariance.

The moment-blindness claim is visible in one dimension. A single Gaussian and a two-mode mixture can share a mean and variance while looking completely different.

FID uses Gaussian fits to feature means and covariances, not feature histograms or pixel-aligned reference pairs.

The phrase "identical FID of zero" is not a hypothetical edge case. Any two distributions with the same first two moments of Inception features score identically under FID, by construction. If one of them happens to place all its mass on a small set of atypical images while the other spreads mass broadly, FID still returns zero. Higher-order structure, all the way down to which specific images appear, is unconstrained.

Three practical issues come with FID.

• Finite-sample estimation. In practice the moments are estimated from finite samples: $\hat\mu_r, \hat\Sigma_r$ from real features and $\hat\mu_g, \hat\Sigma_g$ from generated features. A centered covariance estimate from $n$ vectors in $d$ dimensions has rank at most $\min(d,n-1)$, so it is necessarily rank-deficient when $n\le d$. Chong and Forsyth (CVPR 2020), in "Effectively Unbiased FID and Inception Score and Where to Find Them", show finite-sample, model-dependent FID bias. Rank deficiency alone does not imply a universally downward bias, and using the same sample count does not eliminate differences in estimator bias between models. Report real and generated sample counts and the estimation protocol.
• Encoder dependence. Conventional FID uses Inception-v3 features pretrained for ImageNet classification, not a feature distance fitted directly to human judgments. Domain changes can alter which distinctions the features retain. Validate their relevance before interpreting scores on a new image domain.
• Preprocessing and protocol sensitivity. Image resizing, interpolation, JPEG compression, and whether the model produces the same number of samples as the reference all affect FID. The right discipline is to hold these constant when comparing models and to report them.

A small FID difference may be comparable to estimation noise or model-dependent bias. A bootstrap interval can describe sampling uncertainty, but it does not automatically remove that bias. Resample independent units: clips from the same trajectory may require trajectory-level grouping. Repeated high-dimensional covariance square roots can also be costly. Report the sample counts, estimator, resampling unit, and interval construction.

FVD (Fréchet Video Distance), from Unterthiner et al. (2018), applies the Gaussian feature-distance construction to video clips, using spatiotemporal features such as I3D embeddings. State clip duration, frame rate, spatial resolution, encoder checkpoint, and feature layer. Changes in these can change the score. FVD can detect some temporal failures, but matching its feature moments does not certify object persistence, identity, or causality.

The practical implication is that perceptual and distributional metrics are useful complements to paired-error metrics, not replacements. For either FID or FVD, state the encoder, sample counts, preprocessing, and spatial resolution. For FVD, also state clip duration and frame rate; for FID computed on video frames, state how frames were sampled. Still-image FID has no intrinsic clip-length parameter. When you report LPIPS, state the backbone and the weights. When you report PSNR, state the dynamic range. A single number without its protocol is not a measurement.

A pooled marginal can hide action-conditioning errors. Suppose two actions occur with equal weight and a predictor swaps their distinct future distributions. The pooled mixture is unchanged, so a metric of that marginal cannot detect the swap. A conditional comparison can reveal it if its features and statistic distinguish the two conditionals. Equal pooling weights matter; mirror-image shapes alone do not guarantee equality with unequal action frequencies.

Pooling discards the association between a condition and its output. Stratifying by action or history is one remedy; testing the joint condition-output distribution or using a proper conditional probabilistic score are others. A stratified moment metric still has encoder and moment-matching blind spots.

The next section demonstrates this pooling failure with a one-dimensional toy Gaussian-feature distance, not a published metric. The same loss of conditioning can affect pooled FID or FVD. Paired LPIPS and SSIM retain their specific references, so this argument does not apply to them in the same way.

Unobserved quantities and measurement noise can make a future stochastic given the available history. An exposed point prediction does not communicate that uncertainty, even if the model maintains a distribution internally. To evaluate probabilistic predictions, inspect the declared conditional distribution as well as point-loss performance.

The vocabulary here is easy to get wrong, so let us separate what these words mean when we use them as metrics.

• Visual sharpness: crispness or high-frequency structure in individual rendered samples. This is distinct from forecast sharpness, the concentration of a predictive distribution. A visually sharp sample can come from a broad multimodal forecast.
• Mode coverage: whether the generator represents materially probable alternatives, evaluated with a specified feature-space or event-based diagnostic. Merely assigning nonzero density everywhere is not useful coverage.
• Diversity: the spread of samples the model can generate. High diversity alone is not virtue either; a model that samples noise is diverse but useless.
• Calibration: the agreement between the model's implied probabilities and the empirical frequencies of events it predicts.

These are different properties, and a single number cannot summarize all of them. Image-gradient or frequency diagnostics can describe visual sharpness but also reward noise. Entropy or interval widths can describe forecast concentration under a declared representation. Precision/recall-style diagnostics describe feature-space fidelity and coverage; variance or mode counts describe sampled diversity; event reliability describes calibration. NLL evaluates the probabilities or densities assigned to outcomes, not sharpness alone.

It is worth pausing on why these words are easy to conflate. Visual sharpness is a property of one rendered image. Diversity is a property of a set of rendered images. Mode coverage is a property of the set relative to the true conditional. Calibration is a property of the model's probability assignments, which may or may not be visible in any rendered image. A model can score high on any one of these and low on the others. A table that reports a single number under the heading "quality" is hiding at least three questions.

Best-of-$K$ selection appears in stochastic video-prediction work: the SAVP project presents stochastic predictions selected for their best similarity to the reference among 100 samples. The metric samples $K$ predictions from the model's conditional, compares each to the realized future, and reports the best score. This number improves monotonically as $K$ grows, because you are taking a minimum over more samples; formally, for each realized future $y$ and predicted set $\{\hat y_1, \dots, \hat y_K\}$, the best-of-$K$ error is $\min_{k \in \{1,\dots,K\}} d(\hat y_k, y)$ for a chosen distance $d$, and the expectation of a minimum over $K$ i.i.d. samples is non-increasing in $K$. It is oracle selection: no real deployment gets to see the realized future and pick the closest of $K$ candidates. Best-of-$K$ only has interpretation if $K$ is held fixed across the models you compare and the full sampling protocol is reported.

Adding a sample to a nested pool cannot increase its minimum error. The expectation for i.i.d. samples follows the same ordering. A misspecified forecast can therefore obtain a small best-of-$K$ error if it assigns positive probability to a sufficiently close neighborhood of the reference: the chance of at least one hit is $1-(1-q)^K$, where $q$ is that neighborhood's probability. If $q=0$, no sample budget creates coverage. This improvement is selection, not evidence of calibration.

In the following experiment, each repetition draws one maximum-budget sample pool from a Gaussian centered slightly off the realized future. Each $K$ uses the first $K$ samples from that same pool. These nested sets guarantee non-increasing error for every repetition and for their mean. Independently resampled finite estimates for different $K$ need not be monotone, even though their expectations are non-increasing. Neither result establishes calibration.

The figure is not evidence that the model is getting better. It is evidence that oracle selection is happening. When you see a table that reports best-of-$K$ numbers, check whether the same holdout data is used for all models, whether $K$ is constant, and whether the number is a mean or a minimum. If the sampling protocol is not in the paper, treat the number as uninterpretable.

A deployment may generate one sample or an ensemble. What it cannot generally do is select the sample closest to a future that has not occurred. Report the deployment sample budget and selection procedure separately from reference-based best-of-$K$. Single-sample error and distributional or probabilistic scores are useful complementary diagnostics, not substitutes for that distinction.

High variance need not mean useful diversity. An increasingly wide Gaussian centered between two narrow modes has full real-line support, yet assigns vanishing probability to fixed neighborhoods of those modes as its scale grows. Useful coverage concerns probability mass near plausible alternatives, not merely nonzero density or spread. Feature-space sample diagnostics and event-based probability checks answer different parts of that question.

The reason spread is a weak proxy for coverage is that a distribution can spread out while simultaneously thinning out on the regions that matter.

The same distinction appears as a density plot. The target mixes two Gaussians centered at $-1$ and $+1$, each with standard deviation 0.3. The comparison forecast is a zero-mean Gaussian with standard deviation 2. Its variance is 4, compared with the target's $1+0.3^2=1.09$, so these distributions are not variance-matched. The broad forecast has full support but less density at the target modes, about 26.5% of the target's mode density. We also compute probability mass in the two fixed intervals within 0.3 of each mode: density at a point and probability in a neighborhood are different quantities. The plotted window shows the central region, not the full tails.

As this Gaussian's standard deviation tends to infinity, probability in any fixed bounded neighborhood tends to zero. At every finite positive scale, however, that probability remains positive: samples near the target modes are unlikely rather than impossible. Continuous point probabilities are zero for both distributions; the plotted heights are densities, not point masses.

Sajjadi et al. (2018) formulated generative precision and recall, and Kynkaanniemi et al. (2019) proposed an improved feature-manifold estimator. For that estimator, the two quantities separate:

• Precision: the fraction of generated feature samples inside an estimated real-data support region.
• Recall: the fraction of real feature samples inside an estimated generated-data support region. It is not a fraction of a manifold's geometric volume.

The decomposition is useful because the two failure modes have different fixes. Low precision suggests the generator needs to be constrained toward the real data distribution. Low recall suggests it needs to diversify, for example by using more modes or a less collapsed latent. Reporting a single score hides which of the two problems is present.

These diagnostics are useful because they separate coverage from fidelity, and they avoid the moment-matching limitations of FID. But their semantics depend on hyperparameters (typically $k$-nearest-neighbor radii or manifold balls), on feature choice, and on sample counts. Two implementations with different conventions can rank models differently. Report the implementation you used.

The following example is deliberately simple. It uses one synthesized feature dimension and two discrete actions. We work in one feature dimension with two actions. Under action $a_1$ the true conditional feature is Gaussian with mean $+1$ and standard deviation $0.5$, and under action $a_2$ it is Gaussian with mean $-1$ and standard deviation $0.5$. We have two candidate generators:

• G1 (matched) produces features with mean $+1$ under $a_1$ and mean $-1$ under $a_2$. It matches the ground-truth conditional.
• G2 (reversed) produces features with mean $-1$ under $a_1$ and mean $+1$ under $a_2$. It reverses every action.

With equal action weights, G1 and G2 have identical population marginals: the same two Gaussians appear in the same proportions. We draw their finite samples independently, so the empirical pooled sets and scores will not be identical. Their small differences are sampling effects, not evidence that pooling detects the reversed action labels.

We compute a "toy feature Gaussian distance", defined as the squared 2-Wasserstein distance between two Gaussian fits of the empirical features. For one-dimensional features with means $\mu_x, \mu_y$ and standard deviations $\sigma_x, \sigma_y$,

where:

• $\mu_x$: the sample mean of the first empirical feature set
• $\mu_y$: the sample mean of the second empirical feature set
• $\sigma_x$: the sample standard deviation of the first feature set
• $\sigma_y$: the sample standard deviation of the second feature set

In one dimension, the 2-Wasserstein distance between two Gaussians $\mathcal{N}(\mu_x, \sigma_x^2)$ and $\mathcal{N}(\mu_y, \sigma_y^2)$ reduces to $W_2^2 = (\mu_x - \mu_y)^2 + (\sigma_x - \sigma_y)^2$. The first difference captures a location shift and the second captures a spread mismatch.

The reason we use a Wasserstein distance rather than an $L_2$ distance between densities or parameters is that it is the same construction that FID uses, just with matrices in place of scalars. Keeping the algebra identical while collapsing dimension is what makes the toy example informative: the failure mode you see here is exactly the failure mode a real FID exhibits, minus the high-dimensional bookkeeping.

We compute the distance separately within each action and average with equal weights, then compute it on the concatenated pools. Population pooled distances are both zero. Finite independent samples yield small nonzero pooled distances, while the conditional distance reveals G2's reversal because its action means differ from their references.

This is explicitly a toy feature Gaussian distance. It is not FID, not LPIPS, not SSIM, and not FVD. It is a pedagogical device for the conceptual failure. Real FID and FVD would use high-dimensional learned features and a matrix square root. The algebra is the same and the failure mode is identical.

A reported FID needs its conditioning protocol: a single headline number might be pooled, stratified, or computed for one condition. Pooling alone cannot rule out the balanced label-swap failure. One diagnostic computes FID or FVD within stated action or history strata, then aggregates with declared weights. Multiple covariance estimates and matrix square roots can increase cost, while smaller per-stratum sample sizes increase estimation uncertainty. Stratification does not remove feature-moment blind spots.

Each stratum needs enough samples for reliable statistics. Binning or clustering is one approach for continuous or high-dimensional conditions, but the chosen resolution can merge important differences. Conditional two-sample methods or joint condition-output tests are alternatives. Report the grouping, weights, sample counts, and remaining blind spots.

For two independent samples of size $n\ge2$ from the same nondegenerate Gaussian, the population toy distance is zero but the expected plug-in estimate is positive. Mean and standard-deviation errors are of order $n^{-1/2}$ here. In particular, the expected squared difference of sample means alone is $2\sigma^2/n$. The additional squared standard-deviation difference is nonnegative, so this matched-sample expectation remains positive at finite $n$ and tends to zero as $n$ grows. Degenerate data or comparing a sample with itself are different cases.

The positive expectation arises from squaring estimator differences. It is a property of this plug-in estimator and sampling setup, not a theorem that every finite-sample metric must be biased or that every realization decreases monotonically with sample count.

This is a finite-sample illustration for independent Gaussian samples. It does not establish the magnitude or direction of bias for every real FID/FVD protocol. Real feature covariance estimates additionally become rank-deficient when sample counts are too small for their dimension. Chong and Forsyth (2020) show that FID's finite-sample bias depends on the evaluated generator, so equal sample counts alone do not eliminate biased comparisons. A reported score depends on the model, encoder, sample counts, and estimator.

Using the same sample count for two models does not guarantee equal bias. Report the feature encoder, sampling protocol, estimator, and uncertainty. Confidence intervals describe sampling variation under that protocol; they do not remove estimator bias. Chong and Forsyth's analysis explains why the model-dependent bias also matters when comparing scores.

Negative log-likelihood (NLL) evaluates outcome assignments, not visual sharpness alone. For a declared discrete probability mass function or continuous density $p_\theta$, the realized-future score is

where:

• $\mathrm{NLL}$: the negative log-likelihood of the realized future under the model's predictive distribution (the quantity being defined)
• $p_\theta(o_{t+1:t+H}\mid h_t,a_{t:t+H-1})$: the model's predictive probability (or density) of the realized future observations given history and actions
• $h_t$: the history at time $t$, as defined earlier
• $a_{t:t+H-1}$: the sequence of actions over the prediction horizon

Here the logarithm is natural, so the score is in nats. Lower values indicate more assigned probability mass or density at the realized outcome. With a common reference measure and finite expectations, expected NLL is the true conditional entropy plus the conditional KL divergence, averaged over histories and actions. The true conditional distribution therefore minimizes expected NLL among unrestricted normalized forecasts. Calibration on selected events alone does not establish this optimum.

Zero assigned mass or density gives positive infinity. Discrete NLL is nonnegative; continuous densities can exceed one, giving negative NLL, and their numerical values depend on units and base measure. Clipping a log score is not necessarily scoring a normalized model: flooring the probabilities $(0.9,0.1)$ at $0.2$ gives $(0.9,0.2)$, whose sum exceeds one. Renormalize a floored distribution, or use a declared mixture with a normalized contamination distribution, and disclose the change. A uniform contamination density on an unbounded Euclidean observation space cannot be normalized.

Near-optimal NLL and imperfect event reliability can coexist; exact optimal expected NLL under these assumptions is a stronger statement. For a binary event occurring with probability $0.9$, forecasting $0.85$ gives expected NLL about $0.336$ nats, versus the optimum of about $0.325$ nats, yet the forecast is off by $0.05$. Reliability diagrams reveal that mismatch rather than replacing the log score.

The following reliability curves are stipulated illustrations, not estimates from labeled outcomes. The diagonal represents calibration. The second curve has event frequencies more extreme than the forecast probabilities, so it depicts underconfidence. No average accuracy or NLL is computed for either curve.

For probabilistic forecasts, distinguish concentration of the forecast from its calibration and coverage. Where these diagnostics are defined, report NLL, a diversity diagnostic, and reliability on relevant events. State the interpretation and assumptions of any rank histogram. A sharpness claim alone leaves those other properties untested.

When the target is human perception, direct ratings provide evidence about what the sampled people judge under the tested presentation conditions. They are not an objective certificate of physical correctness. Human studies also have sampling uncertainty and presentation biases, so their protocol belongs beside their scores.

The essentials of a defensible protocol are:

• Rater population and instructions. State whom the study represents, how raters were recruited and instructed, and any quality-screening or exclusion rules. Do not infer valid ratings from agreement alone.
• Blinded, randomized presentation. Hide method identity and randomize candidate order independently of method. A consistency task may need an explicitly identified reference history or action; blinding does not mean hiding information needed to answer the question.
• Separate questions for visual realism and conditional consistency. "Does this look like a real image?" and "Does this prediction correctly continue from the given history under the given action?" measure different things. Ask them separately. A model can be visually realistic but inconsistent, and vice versa.
• Matched playback and sampling. Match horizon, frame rate, resolution, and sample-selection budget when comparing methods on those terms. If presentation is itself an experimental variable, vary it explicitly rather than confounding it with method identity.
• Explicit handling of ties and uncertainty. Choose and report a response format appropriate to the task. A tie or "can't tell" option can express uncertainty; a two-alternative forced-choice design is also legitimate, but it does not directly measure indifference.
• Uncertainty that respects clustering. Frames within a trajectory can be correlated. For inference over independent new trajectories with fixed raters, resample trajectory clusters when within-trajectory dependence is possible. To generalize over raters as well, account for crossed trajectory and rater variation with a suitable resampling scheme or statistical model.
• Reported rater agreement. Use an agreement measure suited to the rating scale and design. Low agreement can reflect ambiguous instructions, heterogeneous preferences, or rater reliability; it does not identify the cause by itself.

These controls address different threats. Blinding reduces identity cues; it cannot guarantee that a method's visual signature is unrecognizable. Separate realism and consistency questions make a trade-off visible rather than merging it into one preference. Matched playback controls presentation differences, while cluster-aware uncertainty avoids treating dependent frames as independent evidence. Agreement statistics describe the ratings, not the physical truth of a continuation.

Report presentation context, including display, audio, color, and playback speed. These can affect judgments. Differences in protocol limit direct comparison between studies unless their effects have been controlled or separately assessed.

Collecting human ratings at scale can be expensive. One alternative is a learned evaluator: a model that predicts a human judgment from an image, a video clip, or a pair of clips. This includes image-aesthetic regressors, video-quality models, video-language models prompted to compare two candidate continuations, and multimodal systems that imitate pairwise comparisons. Within a validated task, such a model can approximate human judgments. Any claimed cost saving needs a separate accounting of repeated inference, training, adaptation, and held-out human validation; agreement alone does not establish a cost ratio. A learned evaluator can also produce scores that reflect the biases of its training set.

Three checks before using a learned evaluator for a reported comparison:

• Held-out human agreement. The evaluator must be shown to agree with held-out human judgments on a dataset that is not part of its training. Training-set agreement alone is not independent validation.
• Domain validation. A judge trained on generic internet video is not guaranteed to give useful scores on surgical footage, satellite imagery, or robot manipulation scenes. Validate on the target domain.
• Specification of prompt and protocol. For a prompted evaluator, report the full prompt, candidate order, temperature, and sampling scheme. Test sensitivity to these choices for the actual judgment task rather than assuming prompt invariance.

There are systemic risks.

• Evaluator overfitting. Training or selecting a generator against a fixed evaluator can exploit its errors. Keep an independent evaluation set and, where possible, independent judges; optimization does not guarantee that a blind spot will be found in every case.
• Shared training bias. A shared pretraining corpus can create correlated blind spots in the generator and evaluator. Do not treat their agreement as independent verification without independent, task-specific validation.
• Adversarial or gaming risk. Learned evaluators may be sensitive to input perturbations or out-of-distribution content. Test robustness relevant to the intended domain rather than assuming that every evaluator has the same vulnerability.

Use a learned evaluator as a measured proxy with a stated validation domain. Held-out human agreement, documented prompts, and matched sample budgets strengthen a comparison but do not guarantee a correct ranking. If the evaluator is used for training, keep that role separate from independent reporting and check whether optimization degrades agreement on held-out judgments.

Learned evaluators are a useful complement to human evaluation, especially for exploring large design spaces. They are not a substitute for it, and they are definitely not a substitute for the fact-based checks that come next. No language model judge is a substitute for verifying that predicted object positions match world coordinates, that velocities are consistent with physics, that interventions produce the right responses, or that the memory state preserves object identity across occlusions.

No single number is a complete evaluation. A defensible scorecard for perceptual and generative performance addresses the following layers with diagnostics suited to the model's outputs and the evaluation task, each reported with its protocol:

• Paired one-step scores. MSE and PSNR with declared dynamic range, on teacher-forced prediction. Report both spatial and per-frame averaging conventions.
• Paired multi-step scores. Same metrics on fixed-action open-loop rollouts, with horizon curves (metric versus $k$) instead of a single pooled number.
• Perceptual metrics. LPIPS with a stated backbone and preprocessing, or SSIM with stated window, Gaussian parameters, and $L$.
• Conditional distribution checks. Where sample support allows it, report FID or FVD within declared action/history strata with explicit aggregation weights, or use another conditional test. Pooled scores can miss reversals; stratified feature moments can also miss conditional differences that preserve those moments.
• Diversity, coverage, and calibration. Use precision/recall-style diagnostics and event reliability where defined. Report realized-future NLL when a normalized mass or density is evaluable under a declared common reference measure. For a sample-only predictor, use a suitable proper sample-based score where its assumptions hold rather than implying that samples expose a likelihood. Disclose normalized smoothing or other forecast modifications; if clipping does not define a normalized forecast, identify the result as a modified log score rather than its NLL.
• Sample-selection protocol. Report $K$ and the selection rule. Future-reference best-of-$K$ is an oracle diagnostic, not deployable selection. A deployment may use one sample, an ensemble, or another rule; evaluate that rule separately.
• Human or learned evaluation. Held-out human agreement for any learned evaluator; blinded randomized protocol for direct human studies.

Discuss score disagreements within their measurement scope. Better pooled FVD but worse action-stratified FVD suggests different aggregate and conditional feature-moment behavior; it does not establish a causal mechanism. Lower test MSE but higher LPIPS means a better empirical squared-error score and a worse feature-dissimilarity score on that protocol, not global optimality or a universal human preference.

Trace each row back to its measured quantity. Pooled FVD summarizes spatiotemporal feature moments over its sample population. Action-stratified FVD compares those moments within the selected action groups. Their disagreement can expose a weakness hidden by pooling, but agreement does not certify complete conditional correctness.

Observation scores can respond to visible consequences of physics or interventions, but do not by themselves certify internal state, persistent memory, or causal correctness. Those checks belong to State, Physics, Causality, and Memory Evaluation. Whether the model supports better decisions is a separate question in Planning, Control, and Policy Evaluation. Datasets, Benchmarks, and Experimental Design covers the broader protocol choices.

Distributional feature metrics allow comparisons between unpaired real and generated sample sets. Learned perceptual metrics provide comparisons calibrated to particular human similarity judgments. Those capabilities complement paired pixel errors; none measures every property of a useful world model.

FID and FVD compare first and second feature moments under a fixed encoder. They miss differences that preserve those moments, including the balanced action reversal in this chapter. LPIPS measures feature similarity calibrated to particular human judgments, not realism in general. SSIM is a local similarity proxy whose window, weights, and dynamic range must be specified. MSE and PSNR can score a stochastic future, but closeness to one realized draw does not identify its conditional distribution. Human ratings measure the sampled judgments under the study conditions; learned judges require their own validation. Neither replaces physical or causal checks.

Optimizing a reported metric does not remove its blind spots. A model designed to minimize FID may match Inception feature moments while predicting the wrong dynamics. A model optimized against a learned evaluator may satisfy that evaluator while violating physics. Report complementary checks for the properties the application needs, and disclose their evaluation cost rather than assuming feature metrics are cheap.

The practical implication is a discipline of reporting. State the protocol, stratify by condition, use horizon curves, hold $K$ fixed, and publish the prompts, backbones, and encoders. Prefer complementary scores over a single headline number. A world model that scores well on every perceptual metric we have discussed and yet moves objects in the wrong direction under a well-specified action is not a good world model, no matter how good the numbers look. The next chapter starts from exactly that point.

Perceptual and generative evaluation answers a specific question: how closely do a model's predicted observations resemble the observations the world produces, individually and in distribution. The chapter developed this question in four layers.

• Reconstruction and predictive accuracy. MSE, PSNR, and SSIM are paired full-reference metrics with different score directions. Expected squared error is minimized by the conditional mean under finite-second-moment assumptions; this theorem does not establish an optimum for expected PSNR or SSIM. In our separated two-mode example the mean looks faded, while a sharp prediction matching the realized mode wins on that draw. Declare dynamic range, averaging conventions, and SSIM preprocessing.
• Perceptual and distributional metrics. LPIPS compares images in the feature space of a pretrained network with weights fit to human judgments, and inherits the backbone, preprocessing, and calibration-set dependencies of that construction. FID is the squared 2-Wasserstein distance between Gaussian fits of Inception features of real and generated distributions; FVD generalizes to spatiotemporal features. Both compare moments, not full distributions, and both depend on encoder choice, sample counts, preprocessing, and spatial resolution. FVD also depends on clip duration and frame rate; video-frame FID needs a declared frame-sampling protocol, not an intrinsic clip-length input.
• Diversity, sharpness, and multimodality. Distributional checks complement point losses for stochastic futures. Nested best-of-$K$ errors cannot increase as $K$ grows, but this oracle rule does not test calibration or deployable selection. High variance is not coverage. Precision/recall depends on its feature-space estimator. Pooling can hide balanced action reversals; stratification exposes our separated-mean example but is not a complete conditional test. Conventional plug-in feature distances can have model-dependent finite-sample bias; sample size, estimator, and uncertainty all matter.
• Human judgment and learned evaluators. Direct ratings require a stated question, presentation and sampling protocol, and uncertainty appropriate to trajectory and rater dependence. Learned evaluators need held-out agreement and domain validation. Their scores remain proxies, particularly when the same judge is used to train or select a generator.

Perceptual and generative scores provide evidence about specified observation properties. They do not certify state, physics, causality, memory, or decision quality. Use them alongside tests designed for those properties. The next chapter examines what the model preserves and predicts beyond visual similarity.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about perceptual and generative evaluation of world models.