Temporal State and Memory › Worked Example: The Dark Corridor

Let us make the corridor concrete and quantitative. We will define a minimal POMDP, then compare two world-model candidates:

• a Markov predictor, an MLP that sees only the current observation and action, and
• a memory-based predictor, a GRU that sees the full history through its recurrent state.

Both are trained on the same objective: predict, at every timestep, the eventual outcome (left or right branch). This is the decision-relevant prediction that the corridor makes hard. Training on it at every timestep, rather than only at the end, matters: it gives the memory model a dense signal for retaining the cue, a point we will return to in the takeaways. Next-observation prediction, remember, is easy for both models here, which is precisely why this task is a good trap for dynamics-only evaluation.

The environment, in detail:

• Underlying state: position along the corridor, plus the hidden branch side.
• Observation: at position $0$, a two-dimensional one-hot cue revealing the branch side. At positions $1$ through $10$, a constant "wall" observation; every reading also carries small independent sensor noise. The observation channel destroys the cue information one step after the entrance.
• Action features: forward during the corridor and a shared terminal decision marker at the end. The code predicts which branch should be chosen; it does not encode the executed left/right choice as an input action.
• Target: the branch side, known from the start of each episode, which the models must predict from the history so far.

We expect the following, and the code will confirm it:

1. The Markov predictor is accurate at step $0$ (the cue is visible) and fluctuates around chance afterward. In the balanced population, the optimal current-input-only accuracy from step 1 onward is $50\%$; a finite held-out sample need not equal exactly 0.5 at every timestep.
2. The GRU maintains near-perfect accuracy throughout, because its recurrent state preserves the cue.
3. A held-out linear probe recovers the branch from the GRU state, while a two-dimensional PCA view shows separated mid-corridor clusters and trajectories.