Interpretability and World-Model Debugging › Interpretability and World-Model Debugging

Imagine you have trained a world model for a small robot. Its one-step prediction loss is low. When you roll the model forward under a fixed action sequence, the predicted trajectories look smooth, physically plausible, and attractive. Then you hand the model to a planner and run the agent. It oscillates, overshoots, drifts into a wall, or settles in the wrong place. Return is poor.

The useful question is where the failure enters the deployed system. In the model-based control pipeline studied here, an encoder or state estimator turns observations into a latent state, a transition model predicts the next latent under an action, a cost model scores outcomes, and a planner searches for actions. Other world models need not have this architecture. Different faults can yield similar trajectories: a biased velocity estimate and an incorrect integration step can both produce drift, while a planner or cost-model change can alter goal-directed behavior. A symptom alone does not identify its source.

Similar symptoms can arise from different components, and one component can produce several symptoms. A low return therefore does not distinguish an estimator fault from a transition or search problem. Controlled experiments can test these explanations. Interface logs, targeted substitutions, and comparisons across rollout regimes provide complementary evidence; no single diagnostic guarantees a unique attribution.

We use two groups of instruments. Representation-level tools include probes for decodability, concept-direction tests for sensitivity of a chosen output, and interventions for response to a specified latent edit. They can be applied at encoder, recurrent, or intermediate network sites, not only inside an encoder. Pipeline-level tools compare teacher-forced, recursive, and closed-loop runs and substitute reference components under disclosed contracts. These tests ask which configuration changes affect the observed behavior, rather than automatically identifying the only faulty component.

Both families are easy to over-read. A probe that recovers velocity tells you the information is accessible to that readout on that distribution. It does not tell you the model uses velocity for control, that the latent is a sufficient Markov state, or that no other information is present. A large directional derivative establishes local output sensitivity to the tested direction; a saliency map needs its own attribution definition before you give it that interpretation. Neither certifies a physical cause in the world. Component substitution measures a behavior difference under a disclosed interface and distribution. It does not automatically identify a unique cause or provide an additive decomposition of error. We will carry these distinctions through the examples.

We build on earlier chapters in Part XI: Evaluation and Understanding. State, physics, causality, and memory evaluation separated observations, latent states, and simulator states. Planning and policy evaluation treated performance as a property of a model-planner-controller combination. Datasets, benchmarks, and experimental design introduced split discipline, matched budgets, paired runs, and reproducibility. We reuse those protocols here. Keep the observation $o_t$, true state $s_t$, estimated or latent state $z_t$, and action $a_t$ separate, including their time indices and coordinate systems.

By the end, you should be able to run a small, fully deterministic diagnostic laboratory on a controlled system, interpret the results, and know which conclusions the experiment does and does not support. That discipline is what motivates the next chapter on failure modes and model exploitation, where we look at what happens when a planner actively searches for the model's blind spots. Once you can localize a fault by hand, you are in a position to understand what it means for an optimizer to find faults for you.