Frontier Questions and Research Roadmaps › Evaluating Beyond Visual Prediction

The third frontier question is about measurement. Auditing whether current metrics support the intended decision claim is a practical research priority proposed here; its payoff depends on the task and available evaluation data.

Part XI introduced five complementary evaluation protocols: teacher-forced one-step prediction, recursive open-loop rollout, intervention-response comparison, closed-loop planning, and transfer. These are not strictly nested tests. Results under one protocol do not by themselves establish performance under another with different inputs or objectives.

These metrics measure different quantities, not progressively more precise versions of one quantity. An image-space loss measures its chosen discrepancies; pixel MSE alone does not establish perceptual fidelity. A calibration metric measures correspondence between stated probabilities or intervals and realized events. A closed-loop metric measures performance on a declared goal. A link between these quantities needs its own assumptions or evidence; a score label does not supply that link.

Consider this conceptual progression of quantities. It is not a universal ordering of evaluation cost or decision relevance:

• One-step prediction loss. Mean squared error or negative log-likelihood of $o_{t+1}$ given the history and action. With densely recorded targets, scoring can be inexpensive relative to new data collection. Its weighting depends on the chosen loss and units; raw observation MSE can be dominated by large actual residuals, not necessarily by the highest-variance coordinates.
• Recursive rollout error. Error after $H$ autoregressive steps with the model consuming its own predictions (Part XII, Chapter 1 analyzed compounding).
• Calibration. Whether stated probabilities or prediction intervals agree with realized events. Calibration is one input to a safety argument, not validation of the complete safety filter; the decision rule and operating assumptions also need testing (Part XII, Chapter 2).
• Interventional prediction error. Error in predicted outcomes under specified interventions, which may be inside or outside the logging policy's support. Such tests directly probe action effects. Other validated prediction and rollout tests can also inform planning; an intervention mean is not an individual counterfactual.
• Action ranking accuracy. Given a state and a set of candidate action sequences, does the model order them the way the environment does? Ranking matters for an argmin over that set. Numerical value scales also matter when a planner applies thresholds, risk penalties or trade-offs between objectives.
• Closed-loop return and success. The decision-relevant outcome, measured in the real or high-fidelity environment, preferably with the evaluator blinded to the model identity where feasible.
• Safety-relevant outcomes. Constraint violations, minimum separation, peak force, recovery latency.
• Latency and adaptation cost. How long a decision takes, and what recovery after a shift requires under a declared criterion. Samples, compute, wall time and energy are possible adaptation-cost measures.

One-step observation loss alone generally does not establish the later outcomes in this list. The next experiment shows why for a particular task. Raw mean squared error weights coordinates equally in the chosen units. High-variance coordinates can dominate when their actual squared prediction errors are large; variance is not an explicit metric weight. A decision depends on the coordinates the task needs, which need not have high variance. More data can reduce prediction errors, even to zero in an appropriate setting, but does not redefine the metric's weighting or supply a universal implication from aggregate error to decision performance. A restricted model family may admit such a link; state and test its assumptions.

Evaluation can depend on the model without being circular. Distinguish population selection from using the model's own predictions as ground truth:

• A policy trained in the model can be evaluated validly in an independent simulator or physical system. The evaluation becomes self-scoring if the learned model also supplies the outcome used as truth.
• Resemblance between logging and evaluation policies does not establish their action-support overlap. Report which states and actions were covered, and which target queries require extrapolation; different policies can also share support.
• If the metric is a learned critic, then a metric improvement can reflect critic overfitting rather than model improvement.

Use an independent outcome evaluator, audit the target state-action coverage, and report prediction metrics alongside realized closed-loop outcomes when assessing a controller. A different collection policy is one possible coverage intervention, not a guarantee of new support. The first experiment uses a task-like evaluation policy distinct from the exploratory training policy. Obtaining additional simulator or hardware evidence has a cost that must be measured for the application.

Model-generated trajectories need not stay in already fitted regions. Scoring them against independent outcomes can reveal errors; reusing the same model as the truth source can instead hide them. Keep tuning instances separate from reporting instances, and document how the evaluation policy selects its population. A fixed independent simulator or physical measurement supplies an outcome check that the learned model does not define.

Two statistical checks matter for interpreting a reported result: overlap between sampling units and the random sources varied by repeated runs.

Leakage across related transitions. Random transition splits can place strongly related samples from one rollout on both sides of the split, making the score optimistic for generalization to new episodes. Correlation alone does not prove memorization, and adjacent samples need not be near-duplicates. Use trajectory-level or episode-level splits when new-episode performance is the question, and task-level splits for new-task performance. Grouping episodes removes within-episode overlap, but shared environments, duplicated trajectories and preprocessing fitted on the test set still require separate checks.

Seeds are not automatically independent systems. Report what each seed varies: initialization, data selection, environment trajectories, task draws or combinations of these. Repeated runs on a fixed physical configuration estimate variability under the random sources actually changed; they do not by themselves sample new arms, camera mounts or floors. If the deployment claim spans those configurations, include representative configurations as sampling units. When only a few were tested, state that limitation without treating seed count as configuration count.

Keep the sampling unit tied to the claim. An error bar over initialization-only repeats answers a different question from one over changed hardware configurations. If evaluation trajectories also vary, the repeat variability includes that randomness. Report the protocol, tested configurations and conditional uncertainty rather than labeling all seed variance as either training noise or deployment risk. There is no general numerical lower-bound relation between these different quantities.

With those principles in hand, we can run three small experiments that make the abstract points concrete.