Failure Modes and Model Exploitation › State Aliasing, Forgetting, and Hallucinated Dynamics

Missing decision-relevant information can cap the best achievable context-dependent performance; it does not make every plan bad. A point prediction can also suffice when an objective depends only on a conditional mean. The cue example isolates information that is needed for the optimal action and absent from the current input. Supplying history or a belief can change the problem, rather than merely searching harder over the same insufficient input.

If two equally likely cue histories produce the same junction input and require opposite actions, a deterministic current-input policy chooses the same action in both. A stochastic policy using only that input and cue-independent randomness has the same conditional action distribution, not necessarily the same realized draw. Either achieves expected reward at most $0.5$ in this binary toy. A policy with a reliably retained cue can achieve $1$.

Consider a two-room environment. At $t = 0$ the agent observes a cue ($\text{A}$ or $\text{B}$) that tells it which room contains the reward. From $t = 1$ onward the agent is at a junction whose observation is identical in both cases. At $t = 2$ the agent must choose "left" or "right"; only the matching action pays reward $1$.

The current-observation-only controller cannot distinguish these cue histories. Feed-forward architecture alone is not the obstruction: a feed-forward function given an explicit history window containing the cue can solve the toy. A one-bit memory can also solve it if the cue is encoded, retained, and read out correctly; nominal recurrent capacity does not guarantee those behaviors.

Paired histories with matching present observations and different action requirements are one useful aliasing test. In a fully specified finite process, representation and transition/reward equivalence can also be checked exhaustively. Passing a sampled suite supports only its tested histories, delays, and metrics; even familiar but untested contexts may fail. Part XI Chapter 3 discusses state, memory, and occlusion diagnostics, not this specific cue-pair protocol. Aliasing and forgetting can coexist, and additional data can improve a learned encoding when the required information is available.

Forgetting is loss of retained information or of previously achieved performance. It need not mean complete ineffectiveness. Separate memory lost during fixed-parameter inference from old-task performance lost after parameter updates; these are different comparisons and can interact.

Within-episode forgetting occurs when information that was encoded is later lost during inference at fixed parameters. A model that never encoded the cue has a different failure. Insufficient long-sequence training or unstable memory can contribute, but neither inadequate training length nor nominal memory capacity determines the outcome. A fixed bit latch retains a cue indefinitely; a four-step input window can instead produce accuracy $1$ at delays zero through three and $0.5$ thereafter. Degradation can therefore be gradual or abrupt. Test delay with fixed parameters rather than identifying the mechanism from curve shape.

Across-update forgetting is deterioration on an old task following new parameter updates under a controlled evaluation protocol. Kirkpatrick et al.'s EWC work (2017) protects important parameters with a quadratic penalty. A decline can be gradual across many optimizer steps or abrupt, and its magnitude can depend on episode length if recurrent dynamics change. Compatible tasks need not cause a decline at all. Compare retained-task performance before and after updates; a mandatory sharp cliff or delay independence is not part of the definition.

Cross two controls: sweep cue delay while parameters are frozen, and compare a fixed retained evaluation set before and after controlled sequential updates. Stratify the latter by delay when relevant. Record encoding, retention, and readout behavior where possible. The scalar experiment below evaluates completed training phases; it does not trace every optimizer step or demonstrate a sharp update-time cliff.

Within-episode retention may benefit from stable explicit memory, appropriate capacity, or longer-sequence training. Across-update retention may benefit from parameter protection, replay, or freezing updates. These interventions are not exclusive: protecting recurrent dynamics can preserve cue retention, and replay can affect both. Diagnose which comparison changed rather than assuming one remedy can tell you nothing about the other mechanism.

The next experiment uses one scalar weight for two incompatible targets on the same inputs: task A predicts $x$, task B predicts $-x$. Its importance weight $q=\operatorname{mean}(x^2)$ is a second-moment/curvature proxy for an EWC-style quadratic anchor, not a computed empirical Fisher. For Gaussian linear regression with known variance $\sigma^2$, expected Fisher is $\mathbb E[x^2]/\sigma^2$. Averaging the conditional Fisher over this toy's fixed input set gives $q/\sigma^2$, hence $q$ at unit variance. Score-squared empirical Fisher on noiseless perfectly fitted data would instead be zero. Keep this likelihood and averaging distinction when interpreting the toy penalty.

With $\lambda_{\rm ewc}=0$, task B is fitted to numerical precision, while retained task-A loss rises to approximately the loss of predicting its negative target. The finite optimizer run does not produce mathematically exact zero B loss. Increasing the penalty lowers the displayed A loss and raises B loss. No single scalar weight gives zero loss on both incompatible targets for nonzero inputs. That impossibility belongs to this one-weight problem; EWC can reduce or avoid forgetting in other settings and is not universally incapable of elimination. The columns loss_A|A, loss_A|B, and loss_B|B compare A immediately after its own training, A after B, and B after B. For the ideal converged quadratic problem with this anchor, the minimizing weight is $(\lambda_{\rm ewc}-1)/(\lambda_{\rm ewc}+1)$; the finite run approximates it. Choosing the penalty allocates error between the incompatible tasks rather than creating capacity to satisfy both.

This measures old-task loss after sequential training on a new target, using the same fixed inputs also used for fitting. It is not a held-out generalization result or a neural-architecture benchmark. Maintain a fixed retained evaluation protocol and check persistent changes across updates, with separate delay tests for inference-time retention. Interventions can overlap; their usefulness must be measured rather than inferred from a mechanism label.

Hallucinated dynamics are model-generated transitions that violate the reference process or a relevant constraint. A state may look plausible in isolation while its transition is impossible; other violations can be obvious from a single state. Plausibility and conditional validity are different checks.

Define a three-state corridor with positions L=$-1$, C=$0$, R=$1$. From C, the reference moves to L or R with probability $0.5$ each. L and R are absorbing: once reached, they remain there. This complete kernel has no return to C. Construct a deterministic model that outputs the reference conditional mean at each of these states. It maps C to C, L to L, and R to R, so its recursive rollout from C stays at C indefinitely. The following code propagates both the full reference kernel and that constructed mean map; no regression fitting is claimed.