Temporal State and Memory › Episodic and External Memory

Recurrent state and attention windows store information automatically. Some tasks call for a store the agent writes to selectively and reads from by content. This section covers two representative instantiations of that third family of memory.

Both recurrent state and context windows update memory automatically from the interaction stream. Recurrence compresses absorbed history into a fixed state, while attention retains every token representation that fits in its finite context. Some tasks instead benefit from a store that the agent writes to selectively and reads by content. Exact soft lookup over $N$ slots costs $O(Nd)$ for slot width $d$, as the denominator below makes visible; approximate indexes can reduce lookup cost at the price of retrieval error and index maintenance. Neural Turing machines and differentiable neural computers formalized differentiable external memory as a matrix $M_t \in \mathbb{R}^{N \times d}$ manipulated by read and write heads:

$$\begin{aligned} m_t &= \sum_{i=1}^{N} w_{t,i}\, M_t[i] \\ w_{t,i} &= \frac{\exp\!\big(k_t^\top M_t[i]\big)}{\sum_{j=1}^{N} \exp\!\big(k_t^\top M_t[j]\big)} \end{aligned}$$

where:

• $M_t$: the memory matrix at time $t$, with $N$ slots
• $M_t[i]$: the contents of slot $i$
• $k_t$: a query vector produced from the current state
• $w_{t,i}$: the attention weight on slot $i$ at time $t$, a softmax over the similarity scores $k_t^\top M_t[j]$ between the query and every slot
• $m_t$: the retrieved vector, the weighted average of the slots most similar to the query

The model retrieves the stored entries most similar to what it currently needs. The mechanism is content-addressed in the same sense that a lookup table keyed by description is: the address is the content itself, or rather the model's learned encoding of it. Writes are more delicate than reads, because a bad write destroys information irreversibly: the model must decide where to write (allocation, often by usage weights so old entries are not clobbered) and what to write (erase and add vectors, as in the LSTM gating philosophy). The gating philosophy reappears here deliberately: writes are soft, gated modifications rather than hard overwrites, so that gradient descent can learn the write policy.

Soft addressing and allocation can become diffuse or saturated during optimization, so differentiable external-memory systems require careful training and evaluation. The same design pattern, selective writes and content-based reads, also provides a useful way to understand episodic control and retrieval-augmented models, even when they replace differentiable memory heads with concrete data structures and explicit retrieval logic.

Episodic control keeps a database of experienced transitions with their realized returns and answers value queries by lookup rather than by generalization:

$$\hat{Q}(s, a) \approx \text{weighted average of stored returns from the } k \text{ nearest neighbors of } (s, a).$$

where:

• $\hat{Q}(s, a)$: the lookup-based value estimate for the state-action pair $(s, a)$
• $k$: the number of nearest neighbors retrieved from the database of stored transitions
• "stored returns": the returns observed after visiting each stored neighbor, not values generalized by a network

The appeal is immediate reuse: after a high return is stored, the next matching query can consult it without waiting for parameter updates. This differs most sharply from one-step temporal-difference learning, which backs a target through one sampled transition at a time. Multi-step returns, eligibility traces, replay, and model-based planning can propagate credit farther, so episodic lookup is not the only alternative. From our current perspective, the deeper point is that episodic control is memory in the service of decisions: it stores evidence for "what worked here before?" The store need not support world prediction, which is a useful reminder that decision usefulness and predictive fidelity are separate criteria.

The same lookup principle applies to prediction. One concrete retrieval-augmented design fetches the $k$ stored trajectories most similar to a query latent state and conditions its prediction on them. Other systems may retrieve transitions, demonstrations, compressed memories, or documents. Instead of relying solely on fixed parameters to encode every dynamical pattern, the model consults stored experience. This has three attractive properties:

• Non-parametric long-term memory. The store can grow with experience rather than squeezing every pattern into fixed parameters. An entry that remains retained is not overwritten by later parameter updates, but availability still depends on capacity, eviction, durability, key stability, and retrieval quality. A retention policy may itself use frequency or importance.
• Interpretable predictions. A prediction can be accompanied by the retrieved neighbors that support it, which helps debugging and trust. When a rollout looks wrong, you can inspect the trajectories the model used as evidence and ask whether they were appropriate analogies, a level of transparency that opaque parameterized predictions cannot offer.
• Locality assumptions made explicit. The model predicts "the future will resemble the futures of similar past situations." Parametric models also encode generalization assumptions, though not necessarily a nearest-neighbor locality assumption; retrieval makes its similarity rule directly inspectable. When similarity in latent space is a good proxy for dynamical similarity, this works well; when it is not, retrieval confidently retrieves the wrong analogies.

The failure mode deserves emphasis because it is a version of model exploitation that we will meet repeatedly in Part VII. An erroneous or outdated entry can steer the agent toward a repeated action. If the write or update rule then adds duplicates, increases the entry's weight, or suppresses contrary evidence, the loop becomes self-reinforcing; a read-only store does not become more entrenched merely because it was queried. Memory therefore needs write discipline and a policy for invalidating experience after the world changes.