Transformers and Neural State-Space Architectures › Transformers and Neural State-Space Architectures

Imagine a household robot operating in a kitchen. Ten minutes ago, it saw a coffee mug placed behind the microwave. The mug is now out of view. Remembering its location could let the robot fetch it directly; forgetting might force a search. In many partially observable tasks, useful evidence arrives at separated times. A short episode may fit in memory in full, but retaining and consulting an arbitrarily long raw history becomes costly under a finite storage and compute budget.

A recurrent state-space model of the kind we built in the previous chapter carries a fixed-size joint state $(h_t,z_t)$. With finite numerical precision, that state has only finitely many representable values. A task may nevertheless admit a finite-dimensional sufficient statistic, such as a correctly specified Kalman filter's mean and covariance in ideal arithmetic; its numerical implementation is an approximation. The difficulty arises when the retained state merges histories that require different predictions or actions. Contractive linear recurrences attenuate old input contributions; gated and other recurrences can retain or transform them differently. Once a relevant distinction is absent from the state and any external memory, later computation cannot recover it from that state alone.

This chapter studies two architectural families with different memory costs. Transformers retain token representations and use attention to compute content-dependent weighted readouts within the available context. Retention makes access possible; it does not guarantee that the representation preserves a detail or that training learns to retrieve it. Neural state-space layers retain a fixed-size recurrent state, using structured transitions that can be evaluated efficiently. Fixed-coefficient and input-selective variants differ in both their mathematics and implementation. Hybrid sequence architectures combine these mechanisms. We will examine named examples, then compare two small predictors on a controlled task rather than infer a universal ranking.

Throughout, keep our standing distinction in mind: a world model predicts environment-relevant quantities that an agent may use for planning or policy learning. Its performance must be checked under the trajectories where it will be used. Four quality axes help organize that evaluation:

• Predictive fidelity: accuracy of forecasts at specified horizons and under specified conditioning.
• Representation quality: retention of information relevant to the prediction or decision task, not necessarily reconstruction of every physical state variable.
• Uncertainty quality: calibration and informativeness of predictive distributions, not simply a large or small variance.
• Decision usefulness: performance of plans or policies when evaluated in the environment.

These axes are related but not interchangeable. The experiment below measures point-prediction errors and an attention diagnostic. It does not test distant event recall, calibrated uncertainty, or a controller.