LSTM Architecture › The Cell State: An Information Highway

The defining feature of an LSTM is its cell state, a separate memory channel that runs parallel to the hidden state. Think of it as a conveyor belt that carries information through time. Unlike the hidden state in vanilla RNNs, which gets completely rewritten at each timestep through a matrix multiplication and nonlinearity, the cell state can preserve information indefinitely by flowing forward with only element-wise operations applied to it.

To appreciate why this matters, consider what happens inside a vanilla RNN at each step. The hidden state is recomputed as $h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$. Both $W_h$ and tanh are involved in that transformation. Tanh squashes any value into the range $[-1, 1]$, so the magnitude of the hidden state is always bounded. When you backpropagate through this squashing operation over many steps, the gradient of the loss with respect to an early hidden state gets multiplied by the derivative of tanh at each step. The derivative of tanh is bounded by 1, often much smaller for large inputs, so the gradient shrinks exponentially. After 50 or 100 steps, the gradient is effectively zero.

The LSTM cell state bypasses this problem in a fundamental way. Information stored in the cell state passes from one timestep to the next primarily through an additive operation: new information is added to old information, with each term selectively scaled by a gate value. Addition does not shrink gradients the way multiplication through nonlinearities does. The gradient of a sum is just the sum of the gradients, which means gradients can travel backward through the cell state pathway without multiplicative attenuation. This additive gradient path is the mathematical core of why LSTMs work.

It is worth pausing to appreciate how unusual this design was when it was proposed in 1997. The mainstream assumption in neural network research at the time was that computation should proceed through stacked layers of learned nonlinear transformations. Every layer or every timestep would apply a weight matrix and a squashing function, and the network would learn by adjusting those weights. Hochreiter and Schmidhuber challenged this by arguing that a memory system needed a way to copy information forward unchanged, not transform it at every step. Their cell state is essentially a write-once, read-selectively buffer: information can be injected into it, it can persist untouched for many steps, and it can later be read out when needed. This design philosophy would later resurface in the residual connections of deep CNNs and in the skip connections that are essential to modern Transformer architectures.

The two key vectors in an LSTM at timestep $t$ are:

• $C_t$: the cell state, which stores long-term memory and flows through time with minimal modification
• $h_t$: the hidden state, which is the output at each timestep and is derived from the cell state through the output gate

In a vanilla RNN, information at timestep $t$ must pass through every intermediate hidden state to reach timestep $t+k$. Each passage involves a matrix multiplication and a tanh activation, which compresses values and causes gradients to shrink exponentially. After enough steps, the original signal is lost in noise.

The cell state sidesteps this problem. Information stored in the cell state can travel from the beginning to the end of a sequence with only element-wise additions and scalings applied to it. Neither of those operations compounds the gradient shrinkage the way repeated matrix multiplications through tanh do. That additive path through time is the core mathematical reason LSTMs can learn long-range dependencies.