LSTM Gate Equations › The Forget Gate

Imagine you're reading a novel and encounter a new chapter. Some details from previous chapters remain important: the protagonist's name, the central conflict, key relationships. Other details have served their purpose and can fade: the color of a minor character's shirt, the exact wording of a throwaway line. Your brain naturally performs this filtering, retaining what matters while letting irrelevant details slip away.

The forget gate gives an LSTM this same capability. At each time step, it examines the current input and the network's recent processing history, then decides, for each piece of stored information, how much to retain. This isn't a binary keep-or-discard decision but a continuous scaling: some memories are preserved almost entirely, others are dimmed, and some are nearly erased.

The forget gate needs to make its retention decisions based on context. What should be forgotten depends entirely on what the network is currently seeing and what it has recently processed. To capture this context-dependence, the gate combines two sources of information:

1. The current input $\mathbf{x}_t$: What new information is arriving right now?
2. The previous hidden state $\mathbf{h}_{t-1}$: What was the network's recent output, summarizing its processing up to this point?

These two vectors are transformed through learned weight matrices and combined:

$$\mathbf{f}_t = \sigma(\mathbf{W}_f \mathbf{x}_t + \mathbf{U}_f \mathbf{h}_{t-1} + \mathbf{b}_f)$$

where:

• $\mathbf{f}_t \in \mathbb{R}^h$: the forget gate activation vector, one value per cell state dimension
• $\mathbf{W}_f \in \mathbb{R}^{h \times d}$: weight matrix that learns which input features signal "time to forget"
• $\mathbf{U}_f \in \mathbb{R}^{h \times h}$: weight matrix that learns which hidden state patterns signal "time to forget"
• $\mathbf{b}_f \in \mathbb{R}^h$: bias vector that sets the default forgetting behavior
• $\sigma$: the sigmoid activation function

The sigmoid function is the mathematical heart of gating. It converts any real number into a value between 0 and 1, perfect for representing "how much to keep":

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

where:

• $z$: the input value (can be any real number)
• $e^{-z}$: the exponential function, which ensures the output is always positive
• The denominator $1 + e^{-z}$ normalizes the result to fall between 0 and 1

The sigmoid has a characteristic S-shape that makes it ideal for soft decisions. When the pre-activation $z$ is large and positive, $e^{-z}$ approaches 0, so $\sigma(z) \approx 1$, meaning "keep this memory." When $z$ is large and negative, $e^{-z}$ becomes very large, so $\sigma(z) \approx 0$, meaning "forget this." For intermediate values, the sigmoid produces intermediate retention levels, allowing the network to partially dim memories rather than making hard binary choices.

Let's trace through the matrix dimensions carefully. Suppose we have an input dimension $d = 100$ (e.g., 100-dimensional word embeddings) and hidden dimension $h = 256$.

The input transformation $\mathbf{W}_f \mathbf{x}_t$ multiplies a $(256 \times 100)$ matrix by a $(100 \times 1)$ vector, producing a $(256 \times 1)$ vector. The hidden state transformation $\mathbf{U}_f \mathbf{h}_{t-1}$ multiplies a $(256 \times 256)$ matrix by a $(256 \times 1)$ vector, also producing a $(256 \times 1)$ vector. These two vectors are added element-wise along with the bias, then sigmoid is applied element-wise to produce the final $(256 \times 1)$ forget gate vector.

The forget gate bias $\mathbf{b}_f$ is often initialized to a positive value (commonly 1 or 2) rather than zero. This ensures that early in training, the forget gate outputs values close to 1, meaning the network starts by remembering everything. Without this initialization trick, the network might learn to forget too aggressively before it has learned what information is worth keeping.