Failure Modes and Model Exploitation › A Diagnostic Workflow for Untrusted Models

The examples suggest a first-pass diagnostic workflow for messier systems. Its order favors inexpensive checks and reusable records, not a theorem that every step eliminates exactly one cause. Some mechanisms can be present together or mask one another.

Controlled experiments can rule out hypotheses and, with adequate design and causal assumptions, establish positive intervention effects. They do not automatically prove a unique mechanism. The sequence below gathers evidence about the tested failure and records what remains uncertain; it is not an empirical ranking of the most frequent causes.

1. Capture and replay the trajectory. Save the initial state, actions, random-stream state, model and environment versions, configuration, and relevant external or hidden inputs. Attempt replay through the same code path. If it fails, investigate the capture boundary, scheduling, hardware, version drift, and external state rather than concluding a unique cause or abandoning diagnosis. Exact replay supports matched-trace substitutions. When it is unavailable, controlled repeated trials can still estimate failure rates and intervention effects with uncertainty; that is population evidence, not the original trajectory's individual counterfactual.

2. Verify the objective and reward conventions. Audit whether the supplied target expresses the intended task on reference trajectories. Separately compare predicted and reference reward or cost using matched timing, discount, termination, and aggregation. A discrepancy can arise from transition bias, reward-head bias, stochastic outcomes, or a protocol mismatch; it does not alone prove objective misspecification. The double-integrator example changes the target itself with exact dynamics. Reward checks are useful early checks here, not a claimed most-common cause based on an absent incidence study.

3. Compare teacher-forced and recursive prediction. At matched reference states and actions, teacher forcing isolates local discrepancies; recursion measures their propagation along predicted states. Small teacher-forced error with large recursive error motivates a propagation analysis, not a unique diagnosis. Large local error can propagate too: a map with constant bias $0.01$ has one-step discrepancy $0.01$ and discrepancy $1$ after $100$ unit-sensitivity steps. Bias and compounding are not mutually exclusive. Plot traces and applicable bounds in a stated norm; a ratio is undefined when its denominator is zero and does not identify a regime by itself.

4. Inspect selected errors and joint support. Compute predicted and reference outcomes for selected plans with an explicit optimism sign. Inspect visited state-action or history-action combinations, not just separate action ranges: matching every marginal range need not cover their combinations. Compare nested budgets under fixed candidate/scoring conventions and use reference scoring controls. Large off-support optimism is evidence to investigate; exploitation can occur on-support, and protocol mismatches can also vary with budget. A positive slope is neither necessary nor sufficient to uniquely identify exploitation.

5. Substitute reference components under stated controls. Compare one intervention at a time, retaining comparable initial conditions, search budgets, and task conventions where possible. Observe how downstream selected actions and outcomes change; successful substitution need not identify a unique cause:

• Replace surrogate dynamics with validated reference dynamics while holding the reward definition fixed. Improvement supports a dynamics-related contribution in that comparison, but interactions or compensating errors can prevent unique attribution.
• Replace a predicted reward head with reference reward while holding surrogate dynamics fixed. This tests reward prediction. Separately replace a supplied proxy with the intended objective to test specification; those are different substitutions.
• Execute a hand-chosen feasible comparator. Success shows that the reference can achieve that outcome, not that the surrogate is sound. The opening feasible plan succeeds while the same surrogate still mispredicts the shortcut; model and selection defects remain compatible with that result.
• Supply a validated fully observable state or sufficient history in place of the current input. Improvement supports an information or representation-use limitation. Distinguish observation aliasing from encoding, retention, readout, and inference effects before assigning a label.

A substitution needs a trustworthy reference and an understood interface. Simulators do not always expose every relevant state, reward, or hidden process; a validated proxy supports only its established scope. Changing one component can change downstream inputs and decisions. Preventing a failure establishes a tested intervention effect, not its unique cause; persistent failure does not prove every remaining defect is downstream. Several interacting or compensating defects may remain.

6. Retest on diagnostic cases and fresh cases. Cases used to select or tune a repair are no longer an untouched holdout. Retest them to detect regressions, then evaluate a separate representative set with uncertainty and the same declared metric. Improvement on diagnostic cases but not fresh cases warns of overfitting, though sampling variation or shift can also explain it. Improvement in two finite sets is evidence, not proof, of population improvement. Preserve a final test set not used to choose the repair when making a confirmatory claim.

Record supported contributions, excluded hypotheses, tested repairs, and unresolved alternatives. Passing these checks does not certify general reliability. A counterexample can refute a universal guarantee; establishing one requires an argument covering the specified domain, sometimes through exhaustive finite checks or formal assumptions. That is different from a diagnostic checklist, but not universally impossible outside research.