Interpretability and World-Model Debugging › Limitations & Impact

The tools in this chapter are powerful and easy to misuse. Let us go through the main limitations, because knowing them is what separates a diagnostic that informs a fix from a diagnostic that manufactures false confidence.

Probes measure accessibility under a readout and fitting protocol. Held-out decoding supports an association, not unique intrinsic semantics. Specify capacity, preprocessing, splits, and target distribution. Different fitted families can give different results; a failed fitted readout does not establish an upper bound on its family's best achievable predictive performance under the same scoring protocol. On a fixed evaluation set with a higher-is-better score, a fitted family member instead supplies a lower bound on the best family score on those same cases. That is not a population-performance bound without further inference assumptions.

For Hewitt-and-Liang selectivity, use their type-consistent randomized mapping or a justified corresponding control. Such tasks retain selected properties, not all statistical structure. A frame-label shuffle is a different randomized-label baseline. Explain what is matched and compare uncertainty before interpreting the score gap.

A CAV is fitted from labeled examples and can inherit labeling noise or nuisance associations. TCAV measures class-output sensitivity along that direction and aggregates derivative signs over selected class examples. A world-model adaptation needs its own output and aggregation protocol; sensitivity does not certify physical causality.

Interchange interventions test a candidate high-level/network alignment. Search may generate candidates; independent held-out validation helps separate selection from testing. Off-support edits, nuisance changes, coordinate choices, recurrence, and dose limit semantic interpretation. A partial recurrent edit can be inconsistent with a physical input history while remaining a valid internal computational counterfactual.

Teacher-forced, recursive, and closed-loop measurements use different conditioning rules. Teacher forcing resets to reference states, recursion feeds back predicted states under fixed actions, and closed-loop execution selects actions through the controller. Their realized state or action values can coincide in particular models and cases. Recursive error need not grow monotonically. A lower-dimensional projection hides distinctions, and a cluster alone proves neither concept association nor causal use. Interpret each trace with its reference inputs, timing, and error definition.

A replacement measures a configuration effect under a specified interface and distribution. It need not improve performance, identify a unique fault, or supply an optimal bound. Disclose information and compute changes when present. Report episode costs and, where useful, time-indexed counterfactual traces without confusing them with same-state local residuals.

Repeating episodes with frozen components measures conditional episode variation, not variation across retrained models. Inferential intervals require a sampling model and a stated construction; repeated deterministic cases alone do not ensure coverage. To study retraining variation, fit multiple models and evaluate each under a common protocol.

These methods answer complementary questions. Probes quantify readout performance; control tasks contextualize memorization capacity; causal-abstraction tests compare aligned counterfactual computations. Rollout comparisons and replacements test suspected pipeline faults. Their usefulness comes from the specific measurement and controls, not a general promise of unique fault localization.

Treat every diagnostic as a measurement under a protocol. State the protocol, report the controls, and be explicit about the scope. When a probe succeeds, say "velocity is linearly accessible from the frozen representation on held-out episodes." When a substitution restores baseline, say "under this interface and distribution, replacing the faulty component restores baseline cost." When an intervention changes the output, say "changing this coordinate changes the planned action under this dose sweep." Each statement keeps its conclusion within the tested scope.

The next chapter, failure modes and model exploitation, examines a planner searching for optimistic predictions. The planned safe-control chapter will address ways to limit that risk. This is a forward link to its intended topic, not a claim that its currently unwritten methods implement these substitution protocols.