Temporal State and Memory › Recurrent Memory

Recurrent networks are a standard learned answer to the compression problem: a fixed-size state updated one step at a time. This section covers how recurrence works, why plain recurrence can struggle over long horizons, and how gated architectures provide a more direct retention path.

The workhorse solution is a recurrent state: a fixed-size vector $h_t$ updated step by step,

$$h_t = f_\theta(h_{t-1},\, o_t,\, a_{t-1}),$$

where:

• $h_t$: the recurrent state at time $t$, a fixed-size vector summarizing the history
• $h_{t-1}$: the previous recurrent state, carrying everything absorbed so far
• $o_t$: the current observation
• $a_{t-1}$: the action taken at the previous step
• $f_\theta$: a neural network, with parameters $\theta$, that decides how to merge the new observation and the last action into the running summary

Everything the model "knows" about the past lives in $h_t$; the history itself is discarded after being absorbed. This is the compression decision in its purest form: one fixed-size vector stands in for an arbitrarily long past, and the network $f_\theta$ is the sole arbiter of what makes it into the summary. Prediction then proceeds from the recurrent state:

$$\hat{o}_{t+1} = g_\phi(h_t, a_t), \qquad \hat{r}_{t+1} = q_\phi(h_t, a_t),$$

where:

• $\hat{o}_{t+1}$: the predicted next observation (the hat marks a prediction, as opposed to the observed $o_{t+1}$)
• $\hat{r}_{t+1}$: the predicted next reward
• $g_\phi$: the observation head, a network with parameters $\phi$ mapping the recurrent state and a candidate action to the next-observation prediction
• $q_\phi$: the reward head, mapping the same inputs to the predicted reward
• $h_t$: the recurrent state defined above
• $a_t$: the action whose consequences are being predicted

Notice that the prediction heads condition on both the recurrent state and the proposed action. They therefore produce action-conditional predictions, and a planner can compare candidate actions by running the heads. Interpreting those predictions as causal consequences additionally requires the data and model to identify the effects of intervening on the action. A policy or planner can operate on the same $h_t$. This is exactly the structure of the recurrent state-space models we will build in detail in Part V, Chapter 1; here we focus on the memory inside the recurrence, not the full world-model architecture.

Conceptually, a recurrent network is doing online compression: at every step it decides, implicitly, which parts of the new observation to merge into the running summary and which parts of the old summary to discard. The compression is shaped by the training objective and data. When trained for the role, $h_t$ can approximate a belief-like filter, a running integral, or a short-term buffer. The catch is that nothing in the architecture guarantees retention over the task horizon. Gated architectures provide a more direct retention path, but optimization must still learn when to use it.

To see why plain recurrence struggles, consider what training demands. If the correct action at step $T$ depends on a cue at step $0$, then the loss at step $T$ must send gradients back through every intermediate hidden state. Learning to preserve the cue means learning weight values for which the cue's influence survives that chain, and whether such weights are discoverable by gradient descent depends on the geometry of the chain itself. For a simple recurrent update $h_t = \tanh(W h_{t-1} + U x_t)$, the derivative of the final state with respect to the initial state is a product of Jacobians:

$$J_t = \frac{\partial h_t}{\partial h_{t-1}}, \qquad \frac{\partial h_T}{\partial h_0} = J_T J_{T-1} \cdots J_1.$$

where:

• $h_0$: the hidden state at the step where the cue arrives
• $h_T$: the hidden state at the step where the loss is computed
• $J_t = \partial h_t / \partial h_{t-1}$: the Jacobian of recurrent step $t$, the matrix of partial derivatives describing how each component of $h_{t-1}$ influences each component of $h_t$
• $J_T J_{T-1} \cdots J_1$: the ordered product of these $T$ matrices, with the latest step on the left, chaining the cue's influence from step $0$ to step $T$
• $x_t$: the input at step $t$ (for a world model, typically the current observation together with the last action)
• $W$: the recurrent weight matrix, multiplying the previous hidden state $h_{t-1}$
• $U$: the input weight matrix, multiplying $x_t$

For matrix-valued, time-varying Jacobians, growth depends on the product's singular directions as well as the norm of each factor. A useful bound is

$$\left\lVert J_T J_{T-1} \cdots J_1 \right\rVert \leq \prod_{t=1}^{T} \left\lVert J_t \right\rVert.$$

Thus uniformly contractive factors are sufficient for decay, while factors with norm above one make growth possible but do not force it: directions can rotate, contract, or cancel. The scalar constant-gain model below isolates the basic effect. If every step multiplies a scalar signal by $\rho$, its magnitude after $T$ steps is $|\rho|^T$: it decays when $|\rho| < 1$, stays constant when $|\rho| = 1$, and grows when $|\rho| > 1$. Over 50 steps, $0.9^{50} \approx 0.00515$, about 0.515 percent of the initial magnitude.

We can make the scalar approximation concrete with a simple computation. The plot shows decay ($\rho = 0.9$), constancy ($\rho = 1.0$), and growth ($\rho = 1.1$) over 50 steps. It illustrates repeated scalar gain, not the exact behavior of every recurrent Jacobian product.

In this scalar illustration, a ten-percent change around one produces orders-of-magnitude differences over a corridor-length span. Gated additive updates are a widely used way to give optimization a controllable near-identity path. They make long-range retention easier to learn, but they are not necessary: other recurrent parameterizations and explicit memory mechanisms can also preserve long dependencies.

Without such a path, the network may need to learn retention through the same long Jacobian chain that attenuates its training signal.

Long short-term memory (LSTM) networks and gated recurrent units (GRUs) attack the problem by changing the shape of the update. Instead of always overwriting the state through a squashing nonlinearity, they introduce an additive channel guarded by learned gates. The key insight is that an additive update, one that adds new content to old content rather than replacing it, provides a gradient path whose gain can be held near one, so information can flow across many steps without exponential decay. For an LSTM, the core equations are:

$$\begin{aligned} i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) & &\text{(input gate)}\\ f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) & &\text{(forget gate)}\\ \tilde{c}_t &= \tanh(W_c x_t + U_c h_{t-1} + b_c) & &\text{(candidate)}\\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t & &\text{(cell state)}\\ \omega_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) & &\text{(output gate)}\\ h_t &= \omega_t \odot \tanh(c_t) & & \end{aligned}$$

where:

• $x_t$: the input at step $t$, for a world model typically the current observation together with the last action
• $h_{t-1}$: the previous hidden state, the network's working output
• $c_t$: the cell state, the protected long-term memory carried across steps
• $\tilde{c}_t$: the candidate value, a proposal for new information to write into the cell state
• $i_t$: the input gate, controlling how much of the candidate is written into the cell state
• $f_t$: the forget gate, controlling how much of the previous cell state survives
• $\omega_t$: the output gate, controlling how much of the cell state is exposed as the hidden state (written $\omega_t$ rather than the conventional $o_t$ to avoid clashing with the observation symbol used elsewhere in this chapter)
• $h_t$: the new hidden state, the gated readout of the cell state
• $\sigma$: the logistic sigmoid, which maps any real number to $(0, 1)$ so each gate acts as a soft switch
• $\odot$: elementwise multiplication
• $W_i, W_f, W_c, W_o$ and $U_i, U_f, U_c, U_o$: weight matrices acting on the input and on the previous hidden state; $b_i, b_f, b_c, b_o$: their bias vectors

Look at the cell-state update $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$. It is a residual update: when the forget gate $f_t$ is near $1$ and the input gate $i_t$ near $0$, information flows through the direct cell-state branch nearly unchanged. Across several such steps, the direct gain is the product of the forget gates, so quantitative retention still depends on how close each gate remains to one. In the corridor, the desired behavior is to write the cue at step 0, preserve it during the featureless walk, and expose it at the decision point. Because the gates depend on the input and hidden state, gradient-based training can learn a write-and-retain policy, provided the objective supplies a useful signal.

The GRU is a streamlined variant with a single update gate $z_t$ and a reset gate $r_t$:

$$\begin{aligned} z_t &= \sigma(W_z x_t + U_z h_{t-1}) & &\text{(update gate)}\\ r_t &= \sigma(W_r x_t + U_r h_{t-1}) & &\text{(reset gate)}\\ \tilde{h}_t &= \tanh(W x_t + U (r_t \odot h_{t-1})) & &\text{(candidate)}\\ h_t &= (1 - z_t)\, h_{t-1} + z_t\, \tilde{h}_t & &\text{(new state)} \end{aligned}$$

where:

• $z_t$: the update gate, blending how much of the old state is kept versus replaced by the candidate
• $r_t$: the reset gate, controlling how much of the old state the candidate can consult
• $\tilde{h}_t$: the candidate hidden state, the proposed new content
• $W_z, W_r, W$ and $U_z, U_r, U$: weight matrices acting on the input and on the previous state (biases omitted for brevity)
• $\sigma$ and $\odot$: the logistic sigmoid and elementwise multiplication, as in the LSTM equations

Notice that the GRU hidden state itself now has the residual form: the update $h_t = (1 - z_t) h_{t-1} + z_t \tilde{h}_t$ is an interpolation between keeping and replacing, so the direct "keep" branch has gain $1 - z_t$. The GRU merges the LSTM's separate cell and hidden states into one, using three affine gate/candidate groups where the displayed LSTM uses four. Which architecture performs better is task-dependent. The LSTM's output gate controls how much cell content is exposed as $h_t$, but it does not isolate the cell completely: $h_t$ feeds the next step's gates and candidate, so noisy exposed state can still influence $c_{t+1}$ indirectly.

Two training choices quietly influence how good the learned memory is. They are easy to overlook because they do not appear in the architecture diagram:

• Truncated backpropagation through time (TBPTT). Unrolled gradients are cut at a window of length $K$, so a loss cannot assign credit directly through more than $K$ recurrent transitions. This is not a hard bound on forward memory: hidden state can be carried across windows, and shorter-range or auxiliary losses can teach a retention rule that generalizes farther. But a terminal-only 50-step dependency receives no direct cue-to-loss gradient when $K=20$, which can make it difficult to learn. Treat $K$ as a credit-assignment limit and test retention at the horizon the task needs.
• Scheduled or curriculum length. Training first on shorter episodes and then extending the horizon can help optimization. One plausible explanation is that shorter examples first establish useful update behavior, which longer examples then refine. That mechanism is a hypothesis unless an ablation separates it from data ordering, regularization, and other optimization effects.

A related trap: if observations occasionally contain the needed cue deterministically and observably, the model can learn memory. If the cue is buried in noise, or appears only under rare action sequences, the data distribution itself may never teach the gates to open. Memory learning is a joint property of architecture, gradient budget, and data distribution. When a memory model fails, before concluding that the architecture is inadequate, check whether the training signal demanded retention, and check whether the truncation window ever spanned the dependence in question.