Perceptual and Generative Evaluation › Human Judgment and Learned Evaluators

When the target is human perception, direct ratings provide evidence about what the sampled people judge under the tested presentation conditions. They are not an objective certificate of physical correctness. Human studies also have sampling uncertainty and presentation biases, so their protocol belongs beside their scores.

The essentials of a defensible protocol are:

• Rater population and instructions. State whom the study represents, how raters were recruited and instructed, and any quality-screening or exclusion rules. Do not infer valid ratings from agreement alone.
• Blinded, randomized presentation. Hide method identity and randomize candidate order independently of method. A consistency task may need an explicitly identified reference history or action; blinding does not mean hiding information needed to answer the question.
• Separate questions for visual realism and conditional consistency. "Does this look like a real image?" and "Does this prediction correctly continue from the given history under the given action?" measure different things. Ask them separately. A model can be visually realistic but inconsistent, and vice versa.
• Matched playback and sampling. Match horizon, frame rate, resolution, and sample-selection budget when comparing methods on those terms. If presentation is itself an experimental variable, vary it explicitly rather than confounding it with method identity.
• Explicit handling of ties and uncertainty. Choose and report a response format appropriate to the task. A tie or "can't tell" option can express uncertainty; a two-alternative forced-choice design is also legitimate, but it does not directly measure indifference.
• Uncertainty that respects clustering. Frames within a trajectory can be correlated. For inference over independent new trajectories with fixed raters, resample trajectory clusters when within-trajectory dependence is possible. To generalize over raters as well, account for crossed trajectory and rater variation with a suitable resampling scheme or statistical model.
• Reported rater agreement. Use an agreement measure suited to the rating scale and design. Low agreement can reflect ambiguous instructions, heterogeneous preferences, or rater reliability; it does not identify the cause by itself.

These controls address different threats. Blinding reduces identity cues; it cannot guarantee that a method's visual signature is unrecognizable. Separate realism and consistency questions make a trade-off visible rather than merging it into one preference. Matched playback controls presentation differences, while cluster-aware uncertainty avoids treating dependent frames as independent evidence. Agreement statistics describe the ratings, not the physical truth of a continuation.

Report presentation context, including display, audio, color, and playback speed. These can affect judgments. Differences in protocol limit direct comparison between studies unless their effects have been controlled or separately assessed.

Collecting human ratings at scale can be expensive. One alternative is a learned evaluator: a model that predicts a human judgment from an image, a video clip, or a pair of clips. This includes image-aesthetic regressors, video-quality models, video-language models prompted to compare two candidate continuations, and multimodal systems that imitate pairwise comparisons. Within a validated task, such a model can approximate human judgments. Any claimed cost saving needs a separate accounting of repeated inference, training, adaptation, and held-out human validation; agreement alone does not establish a cost ratio. A learned evaluator can also produce scores that reflect the biases of its training set.

Three checks before using a learned evaluator for a reported comparison:

• Held-out human agreement. The evaluator must be shown to agree with held-out human judgments on a dataset that is not part of its training. Training-set agreement alone is not independent validation.
• Domain validation. A judge trained on generic internet video is not guaranteed to give useful scores on surgical footage, satellite imagery, or robot manipulation scenes. Validate on the target domain.
• Specification of prompt and protocol. For a prompted evaluator, report the full prompt, candidate order, temperature, and sampling scheme. Test sensitivity to these choices for the actual judgment task rather than assuming prompt invariance.

There are systemic risks.

• Evaluator overfitting. Training or selecting a generator against a fixed evaluator can exploit its errors. Keep an independent evaluation set and, where possible, independent judges; optimization does not guarantee that a blind spot will be found in every case.
• Shared training bias. A shared pretraining corpus can create correlated blind spots in the generator and evaluator. Do not treat their agreement as independent verification without independent, task-specific validation.
• Adversarial or gaming risk. Learned evaluators may be sensitive to input perturbations or out-of-distribution content. Test robustness relevant to the intended domain rather than assuming that every evaluator has the same vulnerability.

Use a learned evaluator as a measured proxy with a stated validation domain. Held-out human agreement, documented prompts, and matched sample budgets strengthen a comparison but do not guarantee a correct ranking. If the evaluator is used for training, keep that role separate from independent reporting and check whether optimization degrades agreement on held-out judgments.

Learned evaluators are a useful complement to human evaluation, especially for exploring large design spaces. They are not a substitute for it, and they are definitely not a substitute for the fact-based checks that come next. No language model judge is a substitute for verifying that predicted object positions match world coordinates, that velocities are consistent with physics, that interventions produce the right responses, or that the memory state preserves object identity across occlusions.

No single number is a complete evaluation. A defensible scorecard for perceptual and generative performance addresses the following layers with diagnostics suited to the model's outputs and the evaluation task, each reported with its protocol:

• Paired one-step scores. MSE and PSNR with declared dynamic range, on teacher-forced prediction. Report both spatial and per-frame averaging conventions.
• Paired multi-step scores. Same metrics on fixed-action open-loop rollouts, with horizon curves (metric versus $k$) instead of a single pooled number.
• Perceptual metrics. LPIPS with a stated backbone and preprocessing, or SSIM with stated window, Gaussian parameters, and $L$.
• Conditional distribution checks. Where sample support allows it, report FID or FVD within declared action/history strata with explicit aggregation weights, or use another conditional test. Pooled scores can miss reversals; stratified feature moments can also miss conditional differences that preserve those moments.
• Diversity, coverage, and calibration. Use precision/recall-style diagnostics and event reliability where defined. Report realized-future NLL when a normalized mass or density is evaluable under a declared common reference measure. For a sample-only predictor, use a suitable proper sample-based score where its assumptions hold rather than implying that samples expose a likelihood. Disclose normalized smoothing or other forecast modifications; if clipping does not define a normalized forecast, identify the result as a modified log score rather than its NLL.
• Sample-selection protocol. Report $K$ and the selection rule. Future-reference best-of-$K$ is an oracle diagnostic, not deployable selection. A deployment may use one sample, an ensemble, or another rule; evaluate that rule separately.
• Human or learned evaluation. Held-out human agreement for any learned evaluator; blinded randomized protocol for direct human studies.

Discuss score disagreements within their measurement scope. Better pooled FVD but worse action-stratified FVD suggests different aggregate and conditional feature-moment behavior; it does not establish a causal mechanism. Lower test MSE but higher LPIPS means a better empirical squared-error score and a worse feature-dissimilarity score on that protocol, not global optimality or a universal human preference.

Trace each row back to its measured quantity. Pooled FVD summarizes spatiotemporal feature moments over its sample population. Action-stratified FVD compares those moments within the selected action groups. Their disagreement can expose a weakness hidden by pooling, but agreement does not certify complete conditional correctness.

Observation scores can respond to visible consequences of physics or interventions, but do not by themselves certify internal state, persistent memory, or causal correctness. Those checks belong to State, Physics, Causality, and Memory Evaluation. Whether the model supports better decisions is a separate question in Planning, Control, and Policy Evaluation. Datasets, Benchmarks, and Experimental Design covers the broader protocol choices.