Temporal State and Memory › Short-Term Versus Long-Term State

The word memory covers two jobs with different requirements: filtering noisy observations into a current estimate, and retaining older information that still matters. This section separates the two jobs and surveys designs that give each its own mechanism.

The phrase "memory" hides a useful distinction. Short-term state answers: what is happening right now, given that observations are noisy, partial, or delayed? This is a filtering problem, and the recurrent state at short horizons is a learned filter, the amortized cousin of the Bayesian filters of Part II, Chapter 2. Like those filters, its job is to fuse a stream of noisy observations into a current estimate, and like them, it should be judged on the accuracy of that estimate at the present moment. Long-term state answers: what happened in the past that still matters? This is a retention problem, and it has different engineering requirements: persistence, capacity, and addressing. A filter is judged on responsiveness and noise rejection; a retention mechanism is judged on whether it holds content without corruption over long spans. Mechanisms optimized for one are frequently poor at the other.

Conflating the two causes predictable failures, and it is worth spelling out the mechanism behind each:

• A per-step filtering objective can dominate a sparse delayed-retention objective when its aggregate gradient is larger. Loss scales, sampling, architecture, and optimization determine whether this happens, so compare the objectives rather than assuming a GRU will always favor one.
• Conversely, a model with only slow accumulators cannot track fast-moving objects or contacts. If the entire state is heavily damped, the freshest information is smeared across many steps, and the model reacts to where the world was, not where it is.

Practical designs therefore separate timescales deliberately, giving each compartment the write policy and update rate its content requires:

• Context windows. A raw buffer can preserve its stored tensors exactly for the last $L$ steps, then evicts them at a hard cutoff. An attention context keeps projected token representations separately addressable within that span, but encoding and learned retrieval can still lose information.
• Recurrent summaries. A GRU/LSTM state compresses absorbed history into a fixed-size state and forgets according to its learned update. Known running statistics often admit compact recurrent updates; arbitrary historical detail generally does not.
• Persistent slots. Dedicated variables for cue-like quantities that must not decay at all: task identity, goal, accumulated score, "which branch did the sign point to." These are written explicitly and rarely, and the rest of the model reads them as stable context.

The last category anticipates episodic and external memory, and the distinction matters for a reason that will keep recurring in this handbook: different memories have different write policies. Short-term state is written automatically and constantly, every step, no decisions required. Cue state should be written only when something new is detected, and a model that rewrites it every step will eventually overwrite it with the featureless corridor observations. Getting the write policy right is not an implementation detail; it is the difference between memory that preserves what matters and memory that grinds it away.

Recurrent state is memory under write-only pressure: information enters, is compressed, and can only influence the future through whatever survives in the summary vector. If the compression discards something, there is no second chance, because the raw history is gone. Transformers, introduced to world modeling through sequence models over trajectories, take the opposite approach: keep the history and read from it directly. Attention computes

$$\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{Q K^\top}{\sqrt{d}}\right) V.$$

where:

• $Q$: the query matrix, projections of the current token
• $K$: the key matrix, projections of each stored past token
• $V$: the value matrix, the content carried by each stored token
• $QK^\top$: the matrix of dot products between every query and every key, scoring how relevant each stored token is to the current one
• $d$: the key dimension; the factor $\sqrt{d}$ keeps the dot-product scores from growing too large before the softmax
• $\operatorname{softmax}(\cdot)$: converts each row of scores into positive weights that sum to one

The queries, keys, and values are projections of stored token representations (and the current input). Reading the formula operationally: at each step, the model compares its query with every retained key, converts those scores into weights, and retrieves a weighted mixture of the values. A cue that remains encoded in context is directly addressable at step 50; it does not have to traverse 50 recurrent updates. Presence does not guarantee recall, however. The learned projections may discard the cue or the attention weights may fail to select it.

The result is a tradeoff: neither mechanism strictly improves on the other. The figure isolates one part of it: cumulative attention-score work over an autoregressive rollout. With a key-value cache, the new query at step $t$ attends to $t$ stored keys, so one step costs $O(t)$ and the cumulative score work through step $T$ is $O(T^2)$. A recurrent update costs $O(1)$ per step with fixed state width, or $O(T)$ cumulatively. Architecture width, layers, and projection costs are omitted, so the curves show scaling rather than wall-clock ratios.