Perceptual and Generative Evaluation › Perceptual and Generative Evaluation

An action can have more than one plausible future. In the small example below, a blob moves either a short or a long distance to the right. The realized future takes the short step. A sharp prediction of the long step puts the blob in the wrong place; averaging the two possible frames produces a faded, two-blob image with lower squared pixel error than that wrong sharp prediction. Yet the sharp prediction matching the realized short step has zero error and beats both. Which comparison should an evaluation report?

The example separates three questions: does an output look plausible, does it match a particular realized future, and does the predictive distribution match the possible futures and their probabilities? These properties can disagree. A score may provide evidence about more than one property, but no single score discussed here certifies all three. This chapter explains what each score measures and what remains unchecked.

We will compute the pixel scores, but we will not run a human-preference study or a learned perceptual network on this toy. Its faded average illustrates why an expected-error optimum need not resemble a sample. Whether people prefer either output requires a separate experiment. The useful task is to choose scores whose questions match the intended use.

This chapter sits inside Part XI: Evaluation and Understanding, after The World-Model Evaluation Ladder. That ladder distinguishes one-step prediction, fixed-action open-loop rollouts, interventions, closed-loop decisions, and transfer. Here we focus on scoring observations under declared prediction protocols. Observation scores can reveal useful failures, but they cannot by themselves certify state, physics, or causal fidelity. The next chapter develops those diagnostics.

The scope is observation evaluation, not a certificate of useful internal state or decision quality. Pixels, feature distances, likelihoods, and human judgments provide different kinds of evidence. A table of those scores becomes interpretable only when the evaluation protocol is stated.

We begin with a synthetic scene and score hand-designed predictions against its realized future. The example requires neither checkpoints nor network training. We then distinguish the questions answered by MSE, PSNR, and SSIM from those answered by LPIPS, FID, and FVD. For stochastic futures, we examine conditional scoring, future-reference best-of-$K$ selection, and an action reversal that pooling conceals. Human judgment protocols and learned evaluators complete the scorecard.

A 48 by 48 grayscale canvas lets us inspect the pixels and recompute the scores. Its two equally probable futures illustrate the difference between matching one realization and describing a distribution. The example does not reproduce the architecture, data, or failure modes of a large video model.

As in Chapter 61, keep three prediction regimes separate in your head:

• Teacher-forced one-step prediction: the model receives the realized observation or history and the action needed to predict the next observation. Errors from its earlier predictions are not fed back as context.
• Fixed-action open-loop rollout: the model receives $h_t$ and actions $a_{t:t+H-1}$ and predicts $o_{t+1:t+H}$ without seeing intermediate realized observations. Earlier prediction errors can affect later predictions, although error need not increase monotonically.
• Closed-loop decision: an agent uses the model inside a planner or policy and receives environment feedback. Task return evaluates decision usefulness; image scores remain diagnostic. Planning, Control, and Policy Evaluation covers that regime.

An MSE of 0.003 needs context: which regime produced it, how many steps were rolled out, and what was averaged? Changing the regime can change the score substantially even with the same model. Keep the regimes separate when reading a prediction table.

The following curves are prescribed illustrations, not measurements from a trained predictor. We construct a gently increasing teacher-forced curve and an exponentially increasing open-loop curve to illustrate a possible difference between protocols. The teacher-forced curve is not constant, and neither shape is a universal law.

Everything we build below uses the convention $p_\theta(o_{t+1:t+H} \mid h_t, a_{t:t+H-1})$, where:

• $o_{t+1:t+H}$: the sequence of future observations the model predicts, from time $t+1$ through time $t+H$
• $h_t = (o_{\le t}, a_{<t})$: the history, consisting of observations up to and including time $t$ and actions strictly before time $t$
• $a_{t:t+H-1}$: the sequence of actions from time $t$ to time $t+H-1$, one action per predicted transition
• $H$: the prediction horizon, i.e. the number of future frames or steps predicted
• $\theta$: the model parameters

History can supply predictive information missing from a single frame in partially observed settings. It is not, by itself, a definition of a world model or a guarantee that the history is sufficient. Supplying actions lets us ask conditional prediction questions; observational action conditioning alone does not identify the effects of causal interventions.

Use $H_{\mathrm{img}}$ for image height and reserve $H$ for the prediction horizon. Grayscale frames have shape $(H_{\mathrm{img}},W)$ and color frames $(H_{\mathrm{img}},W,C)$, with values in $[0,1]$. Here $W$ is width and $C$ is the channel count. Other layouts, such as channels-first, change the axes to average but not the metric's meaning. A rollout adds a time dimension whose length is $H$.