LSTM Gradient Flow › Peephole Connections

In the standard LSTM, the gates (forget, input, output) are computed using only the previous hidden state $\mathbf{h}_{t-1}$ and the current input $\mathbf{x}_t$. They cannot directly observe the cell state $\mathbf{c}_{t-1}$ or $\mathbf{c}_t$. Gers and Schmidhuber (2000) proposed peephole connections as an extension that gives gates direct access to the cell state.

With peephole connections, the forget, input, and output gate equations become:

$$\mathbf{f}_t = \sigma(\mathbf{W}_f \mathbf{x}_t + \mathbf{U}_f \mathbf{h}_{t-1} + \mathbf{V}_f \odot \mathbf{c}_{t-1} + \mathbf{b}_f)$$ $$\mathbf{i}_t = \sigma(\mathbf{W}_i \mathbf{x}_t + \mathbf{U}_i \mathbf{h}_{t-1} + \mathbf{V}_i \odot \mathbf{c}_{t-1} + \mathbf{b}_i)$$ $$\mathbf{o}_t = \sigma(\mathbf{W}_o \mathbf{x}_t + \mathbf{U}_o \mathbf{h}_{t-1} + \mathbf{V}_o \odot \mathbf{c}_t + \mathbf{b}_o)$$

where:

• $\mathbf{V}_f, \mathbf{V}_i \in \mathbb{R}^h$: peephole weight vectors for forget and input gates (connected to $\mathbf{c}_{t-1}$)
• $\mathbf{V}_o \in \mathbb{R}^h$: peephole weight vector for output gate (connected to $\mathbf{c}_t$)
• All $\odot$ represent element-wise multiplication, making these diagonal weight connections

Note that the forget and input gates see $\mathbf{c}_{t-1}$ (the previous cell state), while the output gate sees $\mathbf{c}_t$ (the current cell state, available after the cell state update). This asymmetry is intentional.

Peephole connections change the gradient calculation because now the gate values depend on the cell state. The full partial derivative $\partial \mathbf{c}_t / \partial \mathbf{c}_{t-1}$ (which was approximately $\text{diag}(\mathbf{f}_t)$ in the standard LSTM) now includes additional terms from the peephole connections. The peephole terms add complexity but also allow the network to make gating decisions based on cell state values, potentially enabling more precise control over memory retention. Greff et al. (2017) conducted an extensive ablation study and found that peephole connections provide modest improvements on some tasks but are not always worth the additional complexity.

Peephole connections are most useful for tasks that require precise timing. Consider counting the exact number of steps between two events, or generating periodic patterns at exact intervals. In these cases, the cell state encodes timing information (e.g., a counter), and the gates need to make decisions based on the counter value. Peepholes allow the gate to directly read the counter rather than reading it indirectly through the hidden state.

The distinction matters because the hidden state $\mathbf{h}_{t-1} = \mathbf{o}_{t-1} \odot \tanh(\mathbf{c}_{t-1})$ is a lossy transformation of the cell state: it applies a tanh nonlinearity that compresses values into $[-1, 1]$, and then multiplies by the output gate, which may attenuate the signal further. Reading the cell state directly through peepholes gives access to the uncompressed, unattenuated value.

For most NLP tasks, standard LSTMs without peepholes perform comparably, which is why peephole connections are rarely used in practice today.