Temporal State and Memory › Limitations and Impact

Recurrent and attention-based sequence models are major tools for learned world models, including latent-imagination agents in Part VIII and large video models in Part IX. Learned temporal state is one enabling ingredient among representation learning, optimization, planning algorithms, data, and compute. Earlier systems also handled partial observability through finite histories, frame stacking, finite-state controllers, sampling over histories, and engineered estimators. Modern learned memory reduces some of that engineering burden; it does not replace the other ingredients or every earlier approach.

But the limitations are real, and knowing them will save you weeks of debugging:

• Memory horizons are learned, not guaranteed. TBPTT length $K$ limits direct credit assignment across more than $K$ recurrent steps; it does not impose a hard forward-memory cutoff. Longer retention may emerge from transferable local dynamics, dense losses, auxiliary objectives, or hierarchy, but terminal-only dependencies beyond $K$ are difficult to teach directly. Test memory at the horizon the task requires.
• The bottleneck can lose exactly what matters. Expected training loss weights common examples heavily unless sampling or loss design compensates, so rare decision-critical features can be squeezed out. A planner may then act as if a forgotten hazard were impossible. A calibrated predictive distribution over unresolved alternatives can expose that risk, but only if the model represents the uncertainty rather than collapsing it to a point estimate.
• Long contexts shift the problem rather than solve it. Attention avoids repeatedly compressing all retained tokens into one recurrent vector, but token projections, learned retrieval, finite windows, and cumulative compute still introduce loss and cost. Retrieval adds capacity and inspectability while introducing similarity, retention, staleness, and write-policy failures.
• Evaluation remains unsettled. Short-horizon metrics can hide aliasing, while rollout gaps are not specific to memory. Horizon sweeps, held-out cue probes, controlled history ablations, and task return answer different questions; use them together. We return to this evaluation problem in Part XI.

There is also a deeper conceptual limitation worth sitting with. This chapter emphasizes an individual's interaction history. Abstraction and composition address a complementary source of knowledge: structure shared across situations. Episodic retrieval can transfer from similar episodes, and compositional structure can also help identify the current instance, so the division is not exclusive. Their emphasis differs: memory supports instance-specific continuity, while compositional representations support structural reuse. The next two chapters develop that complementary direction through object-centric and relational representations, then self-models and affordances.