Transformers and Neural State-Space Architectures › Summary

The chapter connects attention-based predictive memory, structured recurrence, and selected hybrids.

• Attention forms query-dependent weighted readouts from retained token representations. Neither the memory nor the readout is necessarily lossless or statistically sufficient. At fixed dimensions, dense sequence attention has quadratic pairwise arithmetic, uncompressed KV storage is linear in context, and cached per-query attention work is linear in context.
• Causal masking supports parallel next-observation supervision only with aligned, leakage-free inputs and targets. Joint action supervision needs an input path that does not already contain the action being predicted. Observational action conditioning requires additional causal and support assumptions before it is used for intervention forecasts.
• Fixed-coefficient linear SSM layers admit recurrent, scan, or convolutional evaluation. Input-selective affine recurrences generally do not have one fixed convolution kernel. Stability claims require the appropriate contraction, forcing, and numerical conditions. HiPPO's optimality concerns its specified weighted projection dynamics, not arbitrary learned fixed recurrences.
• Hybrids combine attention and recurrence or place transformer dynamics inside a stochastic latent world model. Integration versus event retrieval is a useful testing hypothesis, not an exclusive allocation of capabilities.

In the experiment, both models learn finite one-step errors, and recursive errors vary with horizon without a fixed ordering or monotonic-growth guarantee. Unequal capacities and optimization remain confounders. The matched-query attention plot compares learned weights with uniform attention at equal context length; it does not prove a memory mechanism.

The methodological lesson is to compare predictions under the conditioning and actions where they will be used. One-step loss, recursive error, and attention profiles provide complementary diagnostics, not a complete assessment of state representation, uncertainty, or decision usefulness. Those require additional probes, probabilistic evaluation, and controllers tested in the environment. Our short smooth-dynamics task also leaves the opening's distant-event memory problem untested. The next chapter develops autoregressive token world models, including video and action representations, with these distinctions still in force.