Temporal State and Memory › Temporal State and Memory

So far in Part III we have built up a picture of what a world model should represent: observations and sensors in Part III, Chapter 1, sufficient state in Chapter 2, and latent variables in Chapter 3. Each of those chapters answered a question about the content of the model's state at a single moment in time. This chapter addresses a question that is easy to state and surprisingly hard to answer in practice: what should the model's state be when the world's state cannot be recovered from a single observation? This is not an edge case. Many environments an agent cares about, from a robot navigating a building to a language model reading a long document to a game agent with fog of war, hide information from the current observation and store it, instead, in the past.

You already know the general formal answer from Part II: optimal prediction and control under partial observability may require the history of observations and actions, or a task-sufficient state distilled from that history. A Bayesian belief is one such state when the environment model is known. But formal answers do not build themselves. Saying "condition on the history" is a specification, not an implementation. Someone must decide how the history is stored, how far back it reaches, how quickly it can be updated, and what gets forgotten. Those decisions can determine whether an agent can solve the task, and they involve real tradeoffs in computation, memory capacity, and learnability. Those decisions are the subject of this chapter. We will work through recurrent memory, the distinction between short-term and long-term state, episodic and external memory, and the diagnostic failure mode that motivates all of it: state aliasing in non-Markov environments.

Along the way, keep one distinction from Part I, Chapter 2 firmly in view. A world model makes predictions; an agent uses those predictions to act. Memory lives in both places, and the requirements differ. A predictor may need to remember a cue seen a thousand steps ago to forecast an outcome; a controller needs that same memory available now, at decision time, in a form it can act on. This difference in requirements is subtle but consequential. A memory that preserves information faithfully but buries it in a high-dimensional vector that no policy can read from cheaply, or one that is fast to read but discards exactly the facts that distinguish good actions from bad ones, will pass some evaluations and fail others. A memory representation that serves one role but not the other is not yet a world model in the sense of Part I, Chapter 3. Keep asking, as we go: does this memory mechanism serve prediction, decision, or both?