GRU Architecture › The Candidate Hidden State

Before the update gate can blend old and new information, we need to compute the proposed "new" information. The candidate hidden state $\tilde{h}_t$ represents what the hidden state would become if we completely ignored the update gate's blending.

This is where the reset gate comes into play. The candidate computation uses the reset-gated version of the previous hidden state, not the raw hidden state:

$$\tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, x_t] + b_h)$$

where:

• $\tilde{h}_t$: the candidate hidden state, a vector of dimension $h$ with values in $[-1, 1]$
• $r_t$: the reset gate output from the previous computation
• $r_t \odot h_{t-1}$: the reset-gated previous hidden state, where $\odot$ denotes element-wise multiplication
• $[r_t \odot h_{t-1}, x_t]$: concatenation of the reset-gated hidden state and current input
• $W_h$: the candidate's weight matrix of shape $(h, d + h)$
• $b_h$: the candidate's bias vector of dimension $h$
• $\tanh$: hyperbolic tangent activation function, outputting values in $[-1, 1]$

The tanh activation serves two important purposes. First, it bounds the candidate values to a reasonable range $[-1, 1]$, preventing any single dimension from growing unboundedly. Second, it centers the values around zero, which helps with gradient flow during training and allows the hidden state to both increase and decrease in value.

The reset gate's role is clearer now. By gating $h_{t-1}$ before it enters the candidate computation, the reset gate controls how much the previous context influences the new candidate:

• When $r_t \approx 0$: the term $r_t \odot h_{t-1} \approx \mathbf{0}$, so the candidate depends primarily on the current input $x_t$
• When $r_t \approx 1$: the term $r_t \odot h_{t-1} \approx h_{t-1}$, so the candidate incorporates the full previous context

This gives the GRU two distinct ways to control information flow: the reset gate decides how much past context influences the proposal for the new state, while the update gate decides how much of that proposal to accept.