Frontier Questions and Research Roadmaps › Experiment 3: Memory Ablation Under Changing Context

The third experiment compares specified cue-retention and belief-update rules, then changes the goal or corrupts the cue. It is an information-policy comparison, not a pure storage-capacity ablation: the memoryless baseline deliberately ignores the initial cue and later rewards, whereas the updating agent uses both.

To study the benefit of historical information, use a task where the current observation omits relevant information. A fully observed task can still test memory overhead, distraction or harm, but cannot establish that history is necessary. Here the current position omits the hidden goal. The initial cue and later rewards provide information about that goal, and the flip condition changes whether retained cue information remains valid.

The agent controls a scalar position $x_t$ with dynamics

$$x_{t+1} = \mathrm{clip}\!\left( 0.9\, x_t + 0.8\, a_t + \eta_t, \; [-5, 5] \right), \qquad a_t \in [-1, 1]$$

where:

• $x_t$: scalar position at step $t$
• $a_t \in [-1, 1]$: control action at step $t$
• $0.9$: position decay factor
• $0.8$: control gain
• $\eta_t$: process noise at step $t$
• $\mathrm{clip}(\cdot, [-5, 5])$: projection onto the interval $[-5, 5]$

The dynamics coefficients are shared across episodes. What differs is the goal: $x^\star=+1$ in mode 0 and $x^\star=-1$ in mode 1. The non-oracle agents do not observe the mode directly. They can receive an initial cue and subsequent noisy rewards:

• A cue at $t = 0$. A noisy reading of the goal, $\text{cue} = x^\star + \nu$ with $\nu \sim \mathcal{N}(0, 0.7^2)$. For the non-oracle agents, this is the only initial observation carrying goal information; they also observe position.
• A reward signal after each transition. $r_t = -(x_{t+1} - x^\star)^2 + \xi_t$ with $\xi_t \sim \mathcal{N}(0, 1.5^2)$. This is slow and noisy, but it is the only channel that can reveal a change.

The cue supplies evidence before the first action, but is available only at the start and can be corrupted. A cue-only agent that freezes its belief cannot correct it without another signal or update. Rewards arrive after transitions and can support later correction when their goal-conditioned likelihoods differ. At $x_{t+1}=0$ those likelihoods coincide, so that reward alone does not distinguish the two goals. The updating agent combines cue and reward evidence under its assumed model; correction speed depends on the observed evidence and chosen actions.

Part II, Chapter 2's belief-state machinery supplies a useful representation: a distribution $b_t$ over the hidden mode, updated from observations under a specified model. Its adequacy depends on that model and the decision rule. A belief representation alone does not guarantee robustness to noisy cues or changing goals; this experiment tests one particular filter and controller.

Four agents:

• Memoryless. Ignores the cue entirely and starts from a uniform prior $p(m=1) = 0.5$. It uses the prior-mean goal, which is $0$.
• Cue memory, frozen. Uses the cue to set its belief at $t=0$ and never updates it.
• Cue memory with Bayesian update. Uses the cue, then updates the belief from each observed reward, with a hazard rate of $0.05$ representing the prior probability that the context changed on any step.
• Oracle. Knows the mode.

All four share a certainty-equivalent controller: given $p=P(m=1)$, the expected goal is $\hat{x}^\star=1-2p$, and $a=(\hat{x}^\star-0.9x)/0.8$ is clipped to $[-1,1]$. The map from belief to action is fixed; access to information and belief updates differ. This minimizes the one-step expected squared error under the stated prior and zero-mean process noise when clipping is inactive, but it is not a Bayes-optimal multi-step controller. Actions can influence how informative future rewards are, and this controller does not optimize that information-gathering trade-off.

Three conditions exercise the memory in different ways:

• Stable goal, clean cue. The base case. Memory should help.
• Goal flips at $t = 10$. The cue is now stale. A memory that does not update is actively harmful.
• Corrupted cue. Each episode independently has probability 0.5 of reversing the cue-generating mean. The Gaussian cue noise remains unchanged. Under the agents' assumed clean-cue likelihood, a reversed mean can favor the wrong goal, but realized cue direction and confidence vary.

The stable condition compares the specified memories when the goal remains fixed. The flip condition introduces stale goal information; the corrupted-cue condition introduces misleading initial information in a mixture of episodes. These test two failure modes, not an exhaustive memory taxonomy. Stable-condition performance alone does not establish performance under either stress condition.

Episodes are generated with a fixed seed and shared across agents, so exogenous cue and noise draws and the initial state match across agents. Actions, subsequent states, and observed rewards can differ; sharing additive reward noise does not make reward observations identical. This common-random-numbers design aligns exogenous randomness. It can reduce paired-difference variance when the agent costs have positive covariance, but does not remove environment variance or guarantee variance reduction for every pair. Unpaired draws can add comparison noise; whether they do so depends on that covariance. Use the paired episode-cost differences when estimating uncertainty in this matched comparison.

In the stable condition, both cue-based agents have substantially lower mean cost than the memoryless agent. The frozen agent has a slightly lower sample mean than the filter at hazard 0.05, but the reported standard errors do not establish a reliable difference between these two means. The hazard shrinks beliefs toward equal probability; its contribution is better examined with the matched hazard sweep below than attributed from this comparison alone.

The frozen agent has a lower stable-condition sample mean than the filter at hazard 0.05; that is neither a significance result nor an optimality claim. An updating filter with hazard zero can learn from noisy rewards and has lower stable cost in the sweep below. That sweep separates belief updates from the positive-hazard shrinkage toward equal probability: a small positive hazard improves the tested goal-flip mean cost but raises stable-condition mean cost. The chosen update rule and change condition both matter.

In the seeded flip condition, the Bayesian agent has lower mean cost than the frozen agent. The table measures episode cost, not a distribution of detection or recovery times. The frozen agent never updates its cue belief; its mean cost exceeds the memoryless baseline here. This is not a universal ranking of updating and non-updating memories.

The memoryless controller uses an equal-probability goal belief and therefore targets the midpoint. That hedge avoids strong commitment to the wrong endpoint, but also fails to track either endpoint accurately. The frozen controller retains a concentrated, not certain, cue belief. After a change, that concentration can make its actions more costly than the midpoint hedge. A stored estimate can become harmful when its context changes and the controller continues to trust it; the size and direction of the effect depend on the change process and decision rule.

In the corrupted-cue condition, the updating agent has lower mean cost than the frozen agent. Some initial cues favor the wrong goal, with varying confidence. Rewards can then supply evidence for correction when their two goal-conditioned likelihoods differ; at $x_{t+1}=0$ those likelihoods coincide. The selected trace below illustrates correction from a wrong initial belief, not guaranteed recovery in every episode.

This task is a sufficient-statistic toy. The belief is over a two-element set, the update is a one-dimensional filter, and the assumed hazard rate is a single scalar. It is a heuristic prior, not a correctly matched generator: the flip condition changes deterministically at step ten rather than sampling independent flips with probability 0.05. Open-world deployments may involve unknown, high-dimensional or drifting context and an unknown change process, rather than this known binary state space. Other deployments can have small known modes or stationary context. Part XII, Chapter 5 discusses monitoring and correction under changing operating conditions.

The agent knows the two modes and the Gaussian reward-noise model. Its assumed hazard does not match the deterministic flip schedule. In the corrupted-cue condition, the cue likelihood is also misspecified: half the cues come from the opposite goal's Gaussian, but prior_from_cue still assumes the clean-cue model. Recovery here therefore concerns a specified controller and misspecified filter, not a generally correct Bayesian solution. A real deployment can require learning or validating its state space, observation model, change process and update rule separately.

What transfers from the toy is the structural lesson:

1. State explicitly what information the memory is supposed to carry. Here it is "which goal is active."
2. Pair memory with an uncertainty-aware update. Likelihood updates revise an uncertain goal belief as new reward evidence arrives, including when the hazard is zero. A positive hazard separately models possible goal changes by moving the next-decision prior toward equal probability; it is not required for evidence-based updating and does not guarantee better performance.
3. Design the reset boundary. Our episodes reset at $t = 0$. Persisting the previous episode's belief into a new episode can start the controller with misleading evidence when the mode changes. Its effect depends on the initialization and update rule; cross-episode memory need not fail if those rules account for the boundary.
4. Evaluate under change. Report the stable condition and the changed condition. Evaluation only on stable contexts cannot establish recovery after change; it also does not guarantee good stable performance.

The memory experiment above fixed the hazard rate. Sweeping it exposes a non-monotone trade-off. Moving from zero to a small positive hazard sharply lowers cost in the deterministic-flip condition, but increasing it further eventually raises that cost again. Stable-condition cost rises over the tested grid; near 0.5, repeated hedging discards useful accumulated information.