Interpretability and World-Model Debugging › Rollout Visualization and Error Localization

Rollouts compare prediction and execution under specified inputs. Label the regime: teacher-forced prediction, recursive prediction under fixed actions, or closed-loop execution. A number reported merely as rollout error does not tell the reader whether true states were supplied at each step or whether the model's predictions were fed back.

Teacher-forced one-step prediction supplies the true state $s_t$ and recorded action $a_t$ to predict $s_{t+1}$. Each prediction resets to a reference state, preventing feedback of preceding prediction errors. This isolates local transition accuracy at those inputs. Its deployment relevance depends on the available sensing and state-estimation errors; fully observed systems may also have reference-quality state inputs.

Recursive (open-loop) prediction. You start from the true state $s_0$, predict $\hat{s}_1$, then feed $\hat{s}_1$ back into the model to predict $\hat{s}_2$, and so on. Error can compound when earlier prediction discrepancies change later model inputs. Recursive error measures prediction discrepancy at each horizon as model predictions are fed back under a fixed action sequence. That discrepancy may grow, shrink, cancel, or remain zero. It is the natural setting for asking whether the transition model is stable, and it sits between teacher-forced and closed-loop in the sense that it exposes the model to its own errors but still runs under a fixed action sequence.

Closed-loop execution. You run the full agent in the true environment. The planner reads the (possibly corrupted) latent state, chooses an action, the environment transitions, and you observe. Closed-loop behavior measures the decision usefulness of the whole pipeline. It conflates perception, dynamics, and planning, which is exactly why it needs to be decomposed.

These three regimes can disagree sharply. A model with excellent teacher-forced accuracy can have terrible recursive accuracy if it is unstable. A model with good recursive accuracy can have terrible closed-loop behavior if the planner exploits a systematic bias. And a model with mediocre teacher-forced accuracy can have good closed-loop behavior if the planner is robust to the error it makes. In the last case, the model would look bad under one metric and fine under another. Comparing the regimes can reveal this disagreement under the stated inputs and error definitions. It is not the only diagnostic route: direct code inspection, targeted transition tests, or justified structural analysis can also identify specific faults. A regime comparison alone does not establish a unique mechanism.

An aggregate error curve needs inspectable provenance: the inputs, computation, and error definition must be reconstructable. Retained per-step records are one way to provide that provenance; deterministic generation and executable code can provide it too. For detailed fault diagnosis, specify the following fields at each step of each regime:

These are fields to specify in an auditable logging protocol. Subtract states only after aligning time, coordinates, and units. For a learned latent $z$, use a disclosed diagnostic readout $d(z)$ into reference-state coordinates; if no valid readout exists, report a latent-space error against a compatible latent reference instead. A recursive next-state residual includes earlier prediction errors and is not an isolated one-step transition error. In filtering, "innovation" usually means an observation minus its predicted observation; do not silently equate it with a simulator-state residual. The laboratory below stores state and error arrays for its focused comparisons, not a complete implementation of this logging schema.

Compare state-estimation and transition residuals to generate hypotheses, not unique diagnoses. A large estimator residual suggests checking the observation model or estimator. A large teacher-forced transition residual motivates checking the transition implementation, reference states, and timing. Low error on recorded actions with poor closed-loop cost may reflect search, cost specification, state estimation, or errors on planner-selected trajectories absent from those records. Use a controlled substitution or another targeted check to distinguish the explanations.

Trace identity matters too. Every rollout must record which episode, which initial state, which action sequence, which random seeds, and which timestamps it used. Without trace identity, you cannot align a teacher-forced trace with a recursive trace, and you cannot make paired comparisons. A paired difference cancels a shared additive noise term when it enters both outcomes with the same coefficient. With finite outcome variances and the same marginal distributions, positive covariance reduces difference variance relative to independently sampled outcomes. Pairing alone guarantees neither variance reduction nor isolation of a single causal change. Without trace alignment, the intended pairs cannot be reconstructed.

For prediction errors with aligned reference states, horizons, and units, plot error against horizon and overlay teacher-forced and recursive traces, with separate curves for no-fault and faulty conditions. Show closed-loop outcomes separately, as in the laboratory below, unless a common aligned metric has been defined for all regimes. These comparisons make the observed differences visible; they do not identify a unique fault by themselves. A few patterns can guide further checks:

• Flat teacher-forced error with growing recursive error shows accumulation under repeated prediction. It does not by itself prove instability: a constant one-step bias in an integrator also accumulates linearly.
• Growing teacher-forced error shows changing local prediction errors along the recorded trajectory. Systematic bias is one explanation; changing noise, state scales, or input difficulty are others.
• Recursive error non-monotonic. This happens. State error does not have to grow monotonically. A stable model can recover from a perturbation if the dynamics contract toward the true trajectory. A model near the stability boundary can oscillate. Do not assume monotone growth; plot the trace.
• Closed-loop cost high, rollout error low. Possible explanations include inadequate search, model error on planner-selected trajectories absent from the recorded rollouts, state-estimation error, or a cost objective that does not match the intended task. The pattern does not by itself distinguish them.

Low rollout error and high closed-loop cost are not contradictory when the rollout evaluates a different distribution from the planner's choices. Robustness does not require avoiding every state with large model error: some errors are irrelevant to the objective, and some visits are unavoidable. Assess errors on the states and action sequences relevant to the decision, alongside cost and feasibility.

A low-dimensional view of a high-dimensional latent space omits distinctions. PCA plots, nonlinear embeddings, and selected coordinate pairs can reveal patterns, but a visible cluster is not proof of a concept or causal structure. Check labels, episode identities, nuisance variables, and the full-space geometry before interpreting a projected grouping.

A concept-association claim needs evidence beyond a visually appealing cluster, such as held-out label prediction and nuisance controls. A causal-use claim additionally needs a suitable intervention or other causal identification protocol. These are different questions; an intervention is not the only way to establish an association.