Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LSTM Architecture › Gate Mechanism Intuition

Michael BrenndoerferMay 11, 20256 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 77LSTM Architecture › Gate Mechanism Intuition

Reading section 3 of 14 · Gate Mechanism Intuition · 6 minSection 3 of 14 · 6 min

Section 3 of 14 · 6 min read

LSTM Architecture › Gate Mechanism IntuitionOne section · Sections in this chapter3 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

If the cell state flowed forward completely unchanged at every step, the network would have no way to update its memory. LSTMs need a principled way to selectively add new information, remove outdated information, and control what gets exposed as output. They accomplish this through gates, learned sigmoid layers that output values between 0 and 1.

Think of gates as dimmer switches rather than on/off toggles. A gate value of 0.0 means "block everything," 0.5 means "let half through," and 1.0 means "let everything through." Because gates use sigmoid activations, their outputs always fall in the $[0, 1]$ range, making them perfect for scaling information flow. The sigmoid function is differentiable everywhere, which is critical for backpropagation. A hard threshold, where values below 0.5 are set to 0 and values above are set to 1, would create a non-differentiable decision boundary that prevents gradient-based learning.

The gate values are not fixed: they are computed fresh at every timestep by passing the current input and previous hidden state through a small learned network. This means the same LSTM cell can behave differently at different points in a sequence. When processing the beginning of a long document, the forget gate might learn to mostly preserve stored information. When a new named entity appears, the input gate might spike to write it into memory. When a question mark arrives, the output gate might activate to expose accumulated context for an answer prediction. All of this behavior is learned implicitly from training data, not handcrafted by the designer.

Out[5]:

Visualization

S-shaped curve from 0 to 1 over input range -6 to 6 with shaded area under the curve. — Sigmoid activation function outputs values in [0, 1], making it ideal for gates that control information flow. Values near 0 block information while values near 1 pass it through, giving the network a continuous, differentiable way to gate memory.

S-shaped curve from -1 to 1, with positive region shaded green and negative region shaded red. — Tanh activation function outputs values in [-1, 1], allowing candidate values to both increase and decrease cell state elements. The symmetric range around zero enables bidirectional updates to the memory.

LSTMs have three gates, each serving a distinct purpose:

Forget gate: Decides what information to discard from the cell state. When processing a new sentence, the network may want to forget who the previous sentence was about.
Input gate: Decides what new information to store in the cell state. When the model encounters a new subject, it wants to record that subject.
Output gate: Decides what portion of the cell state to expose as the hidden state. Not everything stored in memory is relevant to the current prediction.

Every gate takes the same two inputs: the previous hidden state $h_{t-1}$ and the current input $x_t$ . The general gate computation is:

\text{gate} = \sigma(W \cdot [h_{t-1}, x_t] + b)

where:

$\sigma$ : the sigmoid activation function, which outputs values in $[0, 1]$
$W$ : the weight matrix for this gate (each gate has its own learned weights)
$[h_{t-1}, x_t]$ : the concatenation of the previous hidden state and current input
$b$ : the bias vector for this gate

Because each gate has its own weight matrix, they specialize during training. The forget gate might learn to fire when it sees punctuation marks that signal a sentence boundary; the input gate might learn to fire when it encounters a named entity that needs to be tracked. This specialization is not programmed in: it emerges from gradient descent. What makes the architecture powerful is that the parameterization is rich enough to learn such specialization, and the gating structure provides the mechanism through which that specialization can influence the cell state.

Why Sigmoid and Tanh? A Design ChoiceLink Copied

It is worth asking why the LSTM specifically uses sigmoid for gates and tanh for memory computations. The answer is functional: each activation is chosen for what it computes in context.

Sigmoid maps any real-valued input to the range $(0, 1)$ . When you multiply the cell state element-wise by a vector of sigmoid outputs, each dimension of the cell state gets scaled by a value between 0 and 1. A value of 0 completely erases that dimension; a value of 1 keeps it unchanged; anything in between produces a partial retention. This is exactly the "dimmer switch" behavior that gates require. No other common activation function produces values strictly in $[0, 1]$ in a natural way that is also differentiable and trainable.

Tanh, by contrast, maps inputs to $(-1, 1)$ . This is the appropriate range for candidate values that will be added to the cell state, because a positive candidate increases a dimension's value while a negative candidate decreases it. If you used sigmoid for candidates, you could only ever add positive information to the cell state, never subtract it. The result would be a one-sided accumulation that could not represent the full range of patterns needed in practice. Using tanh allows the cell state to hold both positive and negative magnitudes, which is necessary for representing rich, varied information across different dimensions.

The combination of the two is what makes the full architecture work. The sigmoid gate decides how much to write; the tanh candidate provides what to write. Together they give the network full expressiveness over each cell state dimension.