Perceptual and Generative Evaluation › Diversity, Sharpness, and Multimodality

Unobserved quantities and measurement noise can make a future stochastic given the available history. An exposed point prediction does not communicate that uncertainty, even if the model maintains a distribution internally. To evaluate probabilistic predictions, inspect the declared conditional distribution as well as point-loss performance.

The vocabulary here is easy to get wrong, so let us separate what these words mean when we use them as metrics.

• Visual sharpness: crispness or high-frequency structure in individual rendered samples. This is distinct from forecast sharpness, the concentration of a predictive distribution. A visually sharp sample can come from a broad multimodal forecast.
• Mode coverage: whether the generator represents materially probable alternatives, evaluated with a specified feature-space or event-based diagnostic. Merely assigning nonzero density everywhere is not useful coverage.
• Diversity: the spread of samples the model can generate. High diversity alone is not virtue either; a model that samples noise is diverse but useless.
• Calibration: the agreement between the model's implied probabilities and the empirical frequencies of events it predicts.

These are different properties, and a single number cannot summarize all of them. Image-gradient or frequency diagnostics can describe visual sharpness but also reward noise. Entropy or interval widths can describe forecast concentration under a declared representation. Precision/recall-style diagnostics describe feature-space fidelity and coverage; variance or mode counts describe sampled diversity; event reliability describes calibration. NLL evaluates the probabilities or densities assigned to outcomes, not sharpness alone.

It is worth pausing on why these words are easy to conflate. Visual sharpness is a property of one rendered image. Diversity is a property of a set of rendered images. Mode coverage is a property of the set relative to the true conditional. Calibration is a property of the model's probability assignments, which may or may not be visible in any rendered image. A model can score high on any one of these and low on the others. A table that reports a single number under the heading "quality" is hiding at least three questions.

Best-of-$K$ selection appears in stochastic video-prediction work: the SAVP project presents stochastic predictions selected for their best similarity to the reference among 100 samples. The metric samples $K$ predictions from the model's conditional, compares each to the realized future, and reports the best score. This number improves monotonically as $K$ grows, because you are taking a minimum over more samples; formally, for each realized future $y$ and predicted set $\{\hat y_1, \dots, \hat y_K\}$, the best-of-$K$ error is $\min_{k \in \{1,\dots,K\}} d(\hat y_k, y)$ for a chosen distance $d$, and the expectation of a minimum over $K$ i.i.d. samples is non-increasing in $K$. It is oracle selection: no real deployment gets to see the realized future and pick the closest of $K$ candidates. Best-of-$K$ only has interpretation if $K$ is held fixed across the models you compare and the full sampling protocol is reported.

Adding a sample to a nested pool cannot increase its minimum error. The expectation for i.i.d. samples follows the same ordering. A misspecified forecast can therefore obtain a small best-of-$K$ error if it assigns positive probability to a sufficiently close neighborhood of the reference: the chance of at least one hit is $1-(1-q)^K$, where $q$ is that neighborhood's probability. If $q=0$, no sample budget creates coverage. This improvement is selection, not evidence of calibration.

In the following experiment, each repetition draws one maximum-budget sample pool from a Gaussian centered slightly off the realized future. Each $K$ uses the first $K$ samples from that same pool. These nested sets guarantee non-increasing error for every repetition and for their mean. Independently resampled finite estimates for different $K$ need not be monotone, even though their expectations are non-increasing. Neither result establishes calibration.

The figure is not evidence that the model is getting better. It is evidence that oracle selection is happening. When you see a table that reports best-of-$K$ numbers, check whether the same holdout data is used for all models, whether $K$ is constant, and whether the number is a mean or a minimum. If the sampling protocol is not in the paper, treat the number as uninterpretable.

A deployment may generate one sample or an ensemble. What it cannot generally do is select the sample closest to a future that has not occurred. Report the deployment sample budget and selection procedure separately from reference-based best-of-$K$. Single-sample error and distributional or probabilistic scores are useful complementary diagnostics, not substitutes for that distinction.

High variance need not mean useful diversity. An increasingly wide Gaussian centered between two narrow modes has full real-line support, yet assigns vanishing probability to fixed neighborhoods of those modes as its scale grows. Useful coverage concerns probability mass near plausible alternatives, not merely nonzero density or spread. Feature-space sample diagnostics and event-based probability checks answer different parts of that question.

The reason spread is a weak proxy for coverage is that a distribution can spread out while simultaneously thinning out on the regions that matter.

The same distinction appears as a density plot. The target mixes two Gaussians centered at $-1$ and $+1$, each with standard deviation 0.3. The comparison forecast is a zero-mean Gaussian with standard deviation 2. Its variance is 4, compared with the target's $1+0.3^2=1.09$, so these distributions are not variance-matched. The broad forecast has full support but less density at the target modes, about 26.5% of the target's mode density. We also compute probability mass in the two fixed intervals within 0.3 of each mode: density at a point and probability in a neighborhood are different quantities. The plotted window shows the central region, not the full tails.

As this Gaussian's standard deviation tends to infinity, probability in any fixed bounded neighborhood tends to zero. At every finite positive scale, however, that probability remains positive: samples near the target modes are unlikely rather than impossible. Continuous point probabilities are zero for both distributions; the plotted heights are densities, not point masses.

Sajjadi et al. (2018) formulated generative precision and recall, and Kynkaanniemi et al. (2019) proposed an improved feature-manifold estimator. For that estimator, the two quantities separate:

• Precision: the fraction of generated feature samples inside an estimated real-data support region.
• Recall: the fraction of real feature samples inside an estimated generated-data support region. It is not a fraction of a manifold's geometric volume.

The decomposition is useful because the two failure modes have different fixes. Low precision suggests the generator needs to be constrained toward the real data distribution. Low recall suggests it needs to diversify, for example by using more modes or a less collapsed latent. Reporting a single score hides which of the two problems is present.

These diagnostics are useful because they separate coverage from fidelity, and they avoid the moment-matching limitations of FID. But their semantics depend on hyperparameters (typically $k$-nearest-neighbor radii or manifold balls), on feature choice, and on sample counts. Two implementations with different conventions can rank models differently. Report the implementation you used.

The following example is deliberately simple. It uses one synthesized feature dimension and two discrete actions. We work in one feature dimension with two actions. Under action $a_1$ the true conditional feature is Gaussian with mean $+1$ and standard deviation $0.5$, and under action $a_2$ it is Gaussian with mean $-1$ and standard deviation $0.5$. We have two candidate generators:

• G1 (matched) produces features with mean $+1$ under $a_1$ and mean $-1$ under $a_2$. It matches the ground-truth conditional.
• G2 (reversed) produces features with mean $-1$ under $a_1$ and mean $+1$ under $a_2$. It reverses every action.

With equal action weights, G1 and G2 have identical population marginals: the same two Gaussians appear in the same proportions. We draw their finite samples independently, so the empirical pooled sets and scores will not be identical. Their small differences are sampling effects, not evidence that pooling detects the reversed action labels.

We compute a "toy feature Gaussian distance", defined as the squared 2-Wasserstein distance between two Gaussian fits of the empirical features. For one-dimensional features with means $\mu_x, \mu_y$ and standard deviations $\sigma_x, \sigma_y$,

$$\text{toy-dist}^2\, \;=\; (\mu_x - \mu_y)^2 + (\sigma_x - \sigma_y)^2$$

where:

• $\mu_x$: the sample mean of the first empirical feature set
• $\mu_y$: the sample mean of the second empirical feature set
• $\sigma_x$: the sample standard deviation of the first feature set
• $\sigma_y$: the sample standard deviation of the second feature set

In one dimension, the 2-Wasserstein distance between two Gaussians $\mathcal{N}(\mu_x, \sigma_x^2)$ and $\mathcal{N}(\mu_y, \sigma_y^2)$ reduces to $W_2^2 = (\mu_x - \mu_y)^2 + (\sigma_x - \sigma_y)^2$. The first difference captures a location shift and the second captures a spread mismatch.

The reason we use a Wasserstein distance rather than an $L_2$ distance between densities or parameters is that it is the same construction that FID uses, just with matrices in place of scalars. Keeping the algebra identical while collapsing dimension is what makes the toy example informative: the failure mode you see here is exactly the failure mode a real FID exhibits, minus the high-dimensional bookkeeping.

We compute the distance separately within each action and average with equal weights, then compute it on the concatenated pools. Population pooled distances are both zero. Finite independent samples yield small nonzero pooled distances, while the conditional distance reveals G2's reversal because its action means differ from their references.

This is explicitly a toy feature Gaussian distance. It is not FID, not LPIPS, not SSIM, and not FVD. It is a pedagogical device for the conceptual failure. Real FID and FVD would use high-dimensional learned features and a matrix square root. The algebra is the same and the failure mode is identical.

A reported FID needs its conditioning protocol: a single headline number might be pooled, stratified, or computed for one condition. Pooling alone cannot rule out the balanced label-swap failure. One diagnostic computes FID or FVD within stated action or history strata, then aggregates with declared weights. Multiple covariance estimates and matrix square roots can increase cost, while smaller per-stratum sample sizes increase estimation uncertainty. Stratification does not remove feature-moment blind spots.

Each stratum needs enough samples for reliable statistics. Binning or clustering is one approach for continuous or high-dimensional conditions, but the chosen resolution can merge important differences. Conditional two-sample methods or joint condition-output tests are alternatives. Report the grouping, weights, sample counts, and remaining blind spots.

For two independent samples of size $n\ge2$ from the same nondegenerate Gaussian, the population toy distance is zero but the expected plug-in estimate is positive. Mean and standard-deviation errors are of order $n^{-1/2}$ here. In particular, the expected squared difference of sample means alone is $2\sigma^2/n$. The additional squared standard-deviation difference is nonnegative, so this matched-sample expectation remains positive at finite $n$ and tends to zero as $n$ grows. Degenerate data or comparing a sample with itself are different cases.

The positive expectation arises from squaring estimator differences. It is a property of this plug-in estimator and sampling setup, not a theorem that every finite-sample metric must be biased or that every realization decreases monotonically with sample count.

This is a finite-sample illustration for independent Gaussian samples. It does not establish the magnitude or direction of bias for every real FID/FVD protocol. Real feature covariance estimates additionally become rank-deficient when sample counts are too small for their dimension. Chong and Forsyth (2020) show that FID's finite-sample bias depends on the evaluated generator, so equal sample counts alone do not eliminate biased comparisons. A reported score depends on the model, encoder, sample counts, and estimator.

Using the same sample count for two models does not guarantee equal bias. Report the feature encoder, sampling protocol, estimator, and uncertainty. Confidence intervals describe sampling variation under that protocol; they do not remove estimator bias. Chong and Forsyth's analysis explains why the model-dependent bias also matters when comparing scores.

Negative log-likelihood (NLL) evaluates outcome assignments, not visual sharpness alone. For a declared discrete probability mass function or continuous density $p_\theta$, the realized-future score is

$$\mathrm{NLL} = -\log p_\theta(o_{t+1:t+H}\mid h_t,a_{t:t+H-1})$$

where:

• $\mathrm{NLL}$: the negative log-likelihood of the realized future under the model's predictive distribution (the quantity being defined)
• $p_\theta(o_{t+1:t+H}\mid h_t,a_{t:t+H-1})$: the model's predictive probability (or density) of the realized future observations given history and actions
• $h_t$: the history at time $t$, as defined earlier
• $a_{t:t+H-1}$: the sequence of actions over the prediction horizon

Here the logarithm is natural, so the score is in nats. Lower values indicate more assigned probability mass or density at the realized outcome. With a common reference measure and finite expectations, expected NLL is the true conditional entropy plus the conditional KL divergence, averaged over histories and actions. The true conditional distribution therefore minimizes expected NLL among unrestricted normalized forecasts. Calibration on selected events alone does not establish this optimum.

Zero assigned mass or density gives positive infinity. Discrete NLL is nonnegative; continuous densities can exceed one, giving negative NLL, and their numerical values depend on units and base measure. Clipping a log score is not necessarily scoring a normalized model: flooring the probabilities $(0.9,0.1)$ at $0.2$ gives $(0.9,0.2)$, whose sum exceeds one. Renormalize a floored distribution, or use a declared mixture with a normalized contamination distribution, and disclose the change. A uniform contamination density on an unbounded Euclidean observation space cannot be normalized.

Near-optimal NLL and imperfect event reliability can coexist; exact optimal expected NLL under these assumptions is a stronger statement. For a binary event occurring with probability $0.9$, forecasting $0.85$ gives expected NLL about $0.336$ nats, versus the optimum of about $0.325$ nats, yet the forecast is off by $0.05$. Reliability diagrams reveal that mismatch rather than replacing the log score.

The following reliability curves are stipulated illustrations, not estimates from labeled outcomes. The diagonal represents calibration. The second curve has event frequencies more extreme than the forecast probabilities, so it depicts underconfidence. No average accuracy or NLL is computed for either curve.