GRU Architecture › Summary

This chapter introduced the Gated Recurrent Unit as a streamlined alternative to LSTMs for sequence modeling.

The GRU achieves similar long-range memory capabilities as LSTMs while using a simpler architecture with fewer parameters. The key simplifications are:

• Single state vector: GRUs merge the cell state and hidden state into one, reducing memory overhead and simplifying the information flow.
• Two gates instead of three: The reset gate controls how much past context influences the candidate computation, while the update gate creates a weighted average between old and new states.
• Constrained update: The update gate enforces that the amount "forgotten" and the amount "added" sum to 1, which acts as implicit regularization.

The complete GRU equations, given previous hidden state $h_{t-1}$ and current input $x_t$, are:

1. Update gate: $z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z)$
2. Reset gate: $r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$
3. Candidate: $\tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, x_t] + b_h)$
4. Hidden state: $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$

where $\sigma$ is the sigmoid function, $\tanh$ is the hyperbolic tangent, and $\odot$ denotes element-wise multiplication.

When choosing between GRUs and LSTMs, start with GRUs for their simplicity and efficiency. Switch to LSTMs if you need maximum performance on complex tasks or very long sequences. In practice, the differences are often small enough that other factors matter more.

In the next chapter, we'll explore bidirectional RNNs, which process sequences in both forward and backward directions to capture context from both past and future positions.