Perceptual and Generative Evaluation › Limitations and Impact

Distributional feature metrics allow comparisons between unpaired real and generated sample sets. Learned perceptual metrics provide comparisons calibrated to particular human similarity judgments. Those capabilities complement paired pixel errors; none measures every property of a useful world model.

FID and FVD compare first and second feature moments under a fixed encoder. They miss differences that preserve those moments, including the balanced action reversal in this chapter. LPIPS measures feature similarity calibrated to particular human judgments, not realism in general. SSIM is a local similarity proxy whose window, weights, and dynamic range must be specified. MSE and PSNR can score a stochastic future, but closeness to one realized draw does not identify its conditional distribution. Human ratings measure the sampled judgments under the study conditions; learned judges require their own validation. Neither replaces physical or causal checks.

Optimizing a reported metric does not remove its blind spots. A model designed to minimize FID may match Inception feature moments while predicting the wrong dynamics. A model optimized against a learned evaluator may satisfy that evaluator while violating physics. Report complementary checks for the properties the application needs, and disclose their evaluation cost rather than assuming feature metrics are cheap.

The practical implication is a discipline of reporting. State the protocol, stratify by condition, use horizon curves, hold $K$ fixed, and publish the prompts, backbones, and encoders. Prefer complementary scores over a single headline number. A world model that scores well on every perceptual metric we have discussed and yet moves objects in the wrong direction under a well-specified action is not a good world model, no matter how good the numbers look. The next chapter starts from exactly that point.