Temporal State and Memory › Why Stateless Models Break: A Corridor in the Dark
World Models HandbookAll chapters Temporal State and Memory › Why Stateless Models Break: A Corridor in the DarkOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Imagine an agent entering a long, unlit corridor. At the entrance, it glimpses a sign: an arrow pointing left or right. The sign indicates which branch at the far end of the corridor hides a reward. Then the agent walks forward for fifty steps through a featureless corridor: every observation looks identical. At the end, it must turn left or right.
This scenario is deliberately minimal, and its minimality is what makes it diagnostic. Nothing about it is exotic: no adversarial structure, no complicated dynamics, no partial rewards. The only complication is that one piece of decision-critical information appears once and then vanishes from the observation channel. That is enough to break every stateless model.
During the walk, the agent's current observation carries no information about the correct choice. Yet the correct choice is fully determined by the agent's history, specifically by the sign at step zero. We can make the failure of stateless models precise by splitting the prediction problem into two parts that a stateless model handles very differently:
- Next-observation prediction is easy without memory. In the corridor, the next observation has the same wall mean plus independent sensor noise. A Markov predictor can recover that mean and incur only the irreducible noise error. A dynamics-only evaluation can therefore report low one-step error, and short rollouts may not expose the missing branch cue.
- The realized branch cannot be identified without memory. The current wall observation supports only the prior over branches, which is 50/50 in this example. Expected rewards and values are still well-defined once a future policy and information set are specified, but they average over an unresolved branch rather than identify the episode's correct terminal choice. No amount of processing of the wall observation alone can recover the missing episode-specific cue.
This asymmetry is the central lesson. Partial observability does not always show up as prediction error in the near future. It shows up as irreducible uncertainty about distant outcomes and about which decisions are correct. A model evaluated only on one-step dynamics can be confidently, catastrophically wrong about everything that matters for control. This is one more instance of the recurring theme of this handbook: open-loop prediction quality is not the same as closed-loop decision usefulness. The corridor is the cleanest possible demonstration of that gap. In less stylized systems, the same issue can appear either as an abrupt loss of a cue or as gradual decay across many partially informative observations.
Two underlying situations are aliased when they produce the same observation, or the same observation distribution, but demand different behavior or imply different futures. In the noisy corridor, the two branch states have statistically indistinguishable wall observations but opposite correct terminal choices.
The rest of the chapter is about building state representations that un-alias such situations, and about the engineering tradeoffs involved in doing so at scale. The strategy is always the same in outline: carry forward some trace of the past, and make sure the trace survives until it is needed. The differences among approaches lie in what gets carried, for how long, at what cost, and how reliably the carrying can be learned.