LSTM Gate Equations › The Candidate Cell State

The input gate decides how much to write, but something must decide what to write. This is the role of the candidate cell state: it proposes the actual content that might be added to memory. Think of it as drafting a memo that may or may not be filed, depending on the input gate's judgment.

The candidate needs to be expressive. It should be able to propose increases to certain memory dimensions, decreases to others, and leave some unchanged. This flexibility is necessary because the cell state encodes information through both the magnitude and sign of its values. A sentiment analysis model, for instance, might use positive cell state values to encode positive sentiment and negative values for negative sentiment. The candidate must be able to push the cell state in either direction.

Like the gates, the candidate examines the current input and previous hidden state:

$$\tilde{\mathbf{c}}_t = \tanh(\mathbf{W}_c \mathbf{x}_t + \mathbf{U}_c \mathbf{h}_{t-1} + \mathbf{b}_c)$$

where:

• $\tilde{\mathbf{c}}_t \in \mathbb{R}^h$: the candidate cell state vector, proposing potential updates
• $\mathbf{W}_c \in \mathbb{R}^{h \times d}$: weight matrix that learns what content to extract from the input
• $\mathbf{U}_c \in \mathbb{R}^{h \times h}$: weight matrix that learns how recent processing should shape the update
• $\mathbf{b}_c \in \mathbb{R}^h$: bias vector

Notice the functional difference: the candidate uses $\tanh$ instead of sigmoid. The hyperbolic tangent function is defined as:

$$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$$

where $z$ is the input value. This function outputs values in the range $(-1, 1)$, centered around zero. The centering is key: unlike sigmoid, which only produces positive values between 0 and 1, $\tanh$ can output negative values. This allows the candidate to propose both positive and negative updates to the cell state.

Consider what would happen if the candidate used sigmoid instead. The cell state could only ever increase (since you'd only be adding positive values). Information could accumulate but never be actively reversed. With $\tanh$, the candidate can propose "increase this dimension" (positive values) or "decrease this dimension" (negative values), giving the network the full expressiveness it needs.