GRU Architecture › The Reset Gate: Controlling Past Influence

The reset gate determines how much of the previous hidden state should influence the computation of the new candidate hidden state. Think of it as asking: "How relevant is what I was thinking about to what I'm seeing now?"

When the reset gate outputs values close to 0, the GRU essentially ignores the previous hidden state and computes a new candidate based primarily on the current input. When it outputs values close to 1, the previous hidden state fully participates in computing the candidate. This mechanism allows the GRU to learn when to start fresh versus when to build on past context.

The reset gate computation follows the familiar gating pattern. It takes the previous hidden state and current input, concatenates them, applies a linear transformation, and passes the result through a sigmoid activation:

where:

• $r_t$: the reset gate output, a vector of dimension $h$ with values in $[0, 1]$
• $W_r$: the reset gate's weight matrix of shape $(h, d + h)$, where $d$ is the input dimension and $h$ is the hidden dimension
• $[h_{t-1}, x_t]$: concatenation of previous hidden state and current input, producing a vector of dimension $d + h$
• $b_r$: the reset gate's bias vector of dimension $h$
• $\sigma$: the sigmoid activation function, which squashes any real value to the range $[0, 1]$

The sigmoid activation is necessary here: it ensures every element of $r_t$ lies between 0 and 1, making it suitable for element-wise multiplication as a "soft switch." Values near 0 effectively block information, while values near 1 allow information to pass through unchanged.

The reset gate examines both what the model was thinking about ($h_{t-1}$) and what it's currently seeing ($x_t$) to decide how to weight past information. In language modeling, for instance, the reset gate might learn to output values near 0 when encountering a period or paragraph break, signaling that the previous context is no longer relevant and the model should begin fresh.