LSTM Architecture › LSTM Diagram Walkthrough

Now that the gates make sense individually, let us see how they work together to create a complete memory system. The LSTM processes information in four distinct stages per timestep: decide what to forget, decide what to add, update the cell state, and decide what to output. These four stages happen simultaneously in the equations, but understanding them sequentially builds the right intuition.

Before adding new information, the LSTM first filters out what is no longer relevant. The forget gate examines $h_{t-1}$ and $x_t$, and produces a value between 0 and 1 for each element of the cell state. Elements scaled to near 0 get mostly erased; elements scaled to near 1 get preserved.

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$

where:

• $f_t$: the forget gate output, a vector with one value per cell state dimension
• $W_f$: the forget gate's weight matrix
• $b_f$: the forget gate's bias vector

Consider parsing the sentence "The cat sat on the mat. The dog barked." When the model encounters the period and then "The", the forget gate should fire strongly to clear information about "cat" from the cell state, making room for the upcoming "dog" to be stored as the new subject. The forget gate does this by producing values close to 0 for dimensions that are currently holding subject information, effectively zeroing out that memory slot.

In practice, the forget gate tends to learn values close to 1 for most dimensions most of the time. This is the correct behavior for tasks where context persists across many steps. If the forget gate habitually set values to 0, the LSTM would lose its long-term memory too quickly. The forget gate essentially asks: "Given what I just saw and what I remember, which parts of my memory are no longer needed?" and most of the time the answer is: almost nothing needs to be forgotten yet.

The bias $b_f$ in the forget gate is often initialized to a positive value, such as 1.0 or 2.0. The intuition is that a freshly initialized LSTM should err on the side of remembering rather than forgetting. A large positive bias pushes the sigmoid output close to 1, meaning the network starts with a strong tendency to preserve cell state values. During training, it then learns which contexts warrant forgetting. This initialization trick, popularized by Jozefowicz et al. (2015), improves training stability and long-sequence performance.

With old information selectively cleared, the LSTM decides what to write into memory. This stage involves two parallel computations that work together. The input gate decides which positions in the cell state to update, while a tanh layer computes candidate values that could be written.

$$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$ $$\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C)$$

where:

• $i_t$: the input gate, controlling which cell state positions receive updates
• $\tilde{C}_t$: candidate values that could be added to the cell state
• $W_i, W_C$: separate weight matrices for the input gate and candidate computation
• $b_i, b_C$: corresponding bias vectors

The tanh function is used for the candidates rather than sigmoid because tanh outputs values in $[-1, 1]$, allowing the network to both increase and decrease cell state values. A positive candidate pushes that cell state element upward; a negative candidate pushes it downward. The input gate then scales these proposals, determining how much of each proposed change to apply.

The separation of the input gate from the candidate layer is subtle but important. Why not just compute new values and write them directly? The two-part design allows the network to decouple what kind of information to write from how much of it to write. The candidate layer might compute strong signals for certain dimensions, but the input gate might suppress them because the current context is not appropriate for that kind of update. Conversely, the input gate might activate for dimensions where the candidate values are small but consistent with a persistent pattern the network needs to build up incrementally. This decoupling gives the LSTM a richer vocabulary of memory operations than a single-component update would allow.

Consider what happens when the model encounters the word "not" in the phrase "not happy." The candidate layer might produce a negative value for dimensions associated with positivity, pushing the sentiment representation downward. The input gate activates strongly for those same dimensions to ensure the negation gets applied. Without the separate input gate, the network could not independently control where to update and how much to update.

The actual memory update combines the forgetting and remembering decisions:

$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$$

where:

• $C_t$: the new cell state at timestep $t$
• $C_{t-1}$: the previous cell state from timestep $t-1$
• $f_t$: the forget gate output, scaling old information
• $i_t$: the input gate output, scaling new candidate information
• $\tilde{C}_t$: the candidate values proposed by the tanh layer
• $\odot$: element-wise (Hadamard) multiplication

This formula looks simple but does a lot. The first term, $f_t \odot C_{t-1}$, selectively preserves old information. If $f_t[j] = 0.9$ for some element $j$, we keep 90% of that element's previous value. The second term, $i_t \odot \tilde{C}_t$, selectively writes new information.

The critical observation is that this update is additive, not purely multiplicative. In a vanilla RNN, information at timestep $t$ reaches timestep $t+k$ only through $k$ sequential matrix multiplications. In an LSTM, information stored in the cell state at timestep $t$ can reach timestep $t+k$ through a series of additions and element-wise scalings. The gradient that flows backward through addition does not get multiplied by a small number at each step the way it would through a tanh-saturated weight matrix. This additive pathway is the core mechanism behind LSTMs solving the vanishing gradient problem.

Consider the gradient of the loss $L$ with respect to the cell state at time $t$:

$$\frac{\partial L}{\partial C_t} = \frac{\partial L}{\partial C_{t+1}} \cdot f_{t+1}$$

This gradient flows backward by multiplication with $f_{t+1}$, the forget gate value at the next timestep. If $f_{t+1}$ is close to 1, the gradient passes through almost unchanged. Compare this to a vanilla RNN, where the corresponding gradient involves the derivative of tanh applied to $W_h$, a term that can be very small whenever weights or activations are in the saturating regions of tanh. The LSTM cell state gradient path is far gentler, and a well-trained forget gate that learns to keep values near 1 for important memories creates a near-constant gradient pathway. Hochreiter and Schmidhuber called this the "Constant Error Carousel" (CEC): the cell state can carry an error signal (a gradient) forward in time without it vanishing, as long as the forget gate cooperates.

Let us make this concrete with a numerical example using a 4-dimensional cell state:

After updating the cell state, the LSTM decides what portion of that memory to expose as the current output. The cell state may contain information from many previous steps, but only some of it is relevant to the current prediction. The output gate makes that selection.

$$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$$ $$h_t = o_t \odot \tanh(C_t)$$

where:

• $o_t$: the output gate, deciding which cell state dimensions to expose
• $h_t$: the hidden state, which is the output at this timestep
• $\tanh(C_t)$: the cell state squashed to $[-1, 1]$

The tanh applied to the cell state serves two purposes. First, it bounds the values to $[-1, 1]$, preventing any single dimension from dominating the output. Without this squashing, cell state values could accumulate over many additive updates and grow large in magnitude, causing numerical instability and making the output hard for downstream layers to handle. Second, the tanh re-centers values around zero, which helps downstream computations that typically assume zero-mean inputs.

The output gate then selects which of these bounded values to include in the hidden state. This is where the LSTM distinguishes what it knows and what it reports. The cell state is the network's internal notebook, which may contain many things. The hidden state is what the network decides to share with the world at this particular moment. Suppose the LSTM is generating text and has been tracking the grammatical number (singular or plural) of the current subject. Most of the time that number-tracking information sits quietly in the cell state without being actively used. But when the model reaches a point where a verb needs to agree with the subject, the output gate activates for the dimensions storing number information, pulling it out into the hidden state where it can influence the verb prediction.

This hidden state $h_t$ serves two roles simultaneously: it is the output that downstream layers or classifiers use for predictions, and it is also fed back into the LSTM at the next timestep as $h_{t-1}$. The cell state, by contrast, is internal. It acts as a memory buffer that influences the hidden state through the output gate but is not directly exposed to the rest of the network. This encapsulation is a design virtue: it means the long-term memory (cell state) and the immediate context signal (hidden state) are separate, and the network can develop them differently.

Every timestep runs these six computations in order:

1. Forget gate: $f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$
2. Input gate: $i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$
3. Candidate: $\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C)$
4. Cell update: $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$
5. Output gate: $o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$
6. Hidden state: $h_t = o_t \odot \tanh(C_t)$

Each gate has its own weight matrix and bias, giving the network four parameter groups to learn: $(W_f, b_f)$, $(W_i, b_i)$, $(W_C, b_C)$, and $(W_o, b_o)$. During training, backpropagation adjusts these parameters so the gates learn to open and close at the right times for the task. The precise parameter count and a from-scratch implementation are covered in the LSTM Gate Equations chapter.

Notice that steps 1, 2, 3, and 5 all involve the same inputs ($h_{t-1}$ and $x_t$) processed through different weight matrices. In practice, implementations often batch all four of these computations into a single large matrix multiplication, applying one weight matrix of shape $(4 \times \text{hidden\_size}, \text{input\_size} + \text{hidden\_size})$ at once, then splitting the result into four slices. This is both more efficient on modern hardware and mathematically equivalent to computing each gate separately. PyTorch's nn.LSTM uses this batched formulation internally.