LSTM Architecture › LSTM Variants and Alternatives

The original LSTM architecture spawned a family of related designs that modify the gating structure in different ways. Understanding these variants illuminates which parts of the LSTM architecture are necessary and which admit alternatives.

Standard LSTM gates receive only $h_{t-1}$ and $x_t$ as inputs. A variant proposed by Gers and Schmidhuber (2000) adds "peephole connections" that also feed the cell state directly into the gates:

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t, C_{t-1}] + b_f)$$

The intuition is that the gates should be able to look at the memory contents in addition to the previously exposed hidden state. Consider a clock-like memory: the gate deciding when to reset a counter would benefit from seeing the current count directly, rather than inferring it through the hidden state. Peephole connections add a small number of parameters per gate and can improve performance on tasks requiring precise timing, such as speech recognition with long silence gaps. In practice, the improvement is often modest, and peephole connections are not included in PyTorch's standard nn.LSTM.

The Gated Recurrent Unit (GRU), introduced by Cho et al. (2014), simplifies the LSTM by merging the cell state and hidden state into a single vector and replacing the three-gate system with two gates: a reset gate and an update gate. The GRU has fewer parameters than an LSTM of the same hidden size and tends to train faster. On many tasks, GRUs and LSTMs perform comparably. The choice between them is often empirical. The next chapter covers GRUs in depth.

The LSTM's forget gate and input gate are independent, which means it is possible for both to have high values simultaneously. A coupled variant forces them to be complementary:

$$i_t = 1 - f_t$$

This ensures that the total information in each cell state dimension remains roughly constant: if the forget gate retains 70% of the old value, the input gate contributes exactly 30% of the new candidate. This hard coupling reduces the parameter count and enforces a conservation-like property on the cell state. It works well on some tasks but can be too restrictive for others.