GRU Architecture › Complete GRU Equations

Having explored each component individually, we're now ready to see how they fit together into a unified computational framework. The beauty of the GRU lies not in any single equation, but in how these pieces orchestrate a delicate dance of remembering and forgetting at every timestep.

Imagine you're reading a novel. At each word, your brain performs a remarkable feat: it decides which aspects of the previous context remain relevant, proposes an updated understanding, and then blends this new understanding with what you already knew. The GRU mirrors this process through four sequential computations, each building on the previous.

At each timestep $t$, the GRU receives two inputs: the previous hidden state $h_{t-1}$ (what the model "remembers" from processing earlier timesteps) and the current input $x_t$ (the new information arriving now). From these, it must produce a new hidden state $h_t$ that appropriately blends old context with new observations.

Step 1: Assess how much to update (the update gate)

Before doing anything else, the GRU asks: "Given what I knew and what I'm seeing now, how much should I change my understanding?" The update gate answers this question:

$$z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z)$$

This gate examines both the previous hidden state and current input, then outputs a value between 0 and 1 for each dimension of the hidden state. A value near 0 means "keep the old information," while a value near 1 means "embrace the new." The sigmoid function $\sigma$ ensures these values stay in the valid range.

Step 2: Determine past relevance (the reset gate)

Next, the GRU considers a subtler question: "When computing my proposal for new information, how much should the past influence that proposal?" The reset gate provides this answer:

$$r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$$

This gate has a different purpose than the update gate. While the update gate controls the final blending, the reset gate controls what goes into the "new" option being considered. When the reset gate outputs values near 0, the GRU effectively ignores its memory when formulating a new candidate, allowing it to make a fresh start.

Step 3: Propose new content (the candidate hidden state)

With the reset gate computed, the GRU can now propose what the new hidden state might look like. This candidate incorporates the current input and a reset-modulated version of the previous state:

$$\tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, x_t] + b_h)$$

The notation $r_t \odot h_{t-1}$ represents element-wise multiplication: each dimension of $h_{t-1}$ is scaled by the corresponding dimension of $r_t$. Where the reset gate is low, the previous hidden state's influence is suppressed; where it's high, the previous state participates fully. The tanh activation bounds the candidate to $[-1, 1]$, preventing unbounded growth.

Step 4: Blend old and new (the final hidden state)

Finally, the GRU combines everything using the update gate as a blending coefficient:

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

This formula is a weighted average computed element-wise. For each dimension $i$ of the hidden state:

• The term $(1 - z_t[i]) \cdot h_{t-1}[i]$ represents how much of the old value to keep
• The term $z_t[i] \cdot \tilde{h}_t[i]$ represents how much of the new candidate to add
• Since $(1 - z_t[i]) + z_t[i] = 1$, the weights always sum to one

This constraint is what distinguishes GRUs from LSTMs: you cannot simultaneously keep everything old and add everything new. The architecture forces a trade-off, which reduces flexibility but also reduces parameters and can act as implicit regularization.

Throughout these equations, we use consistent notation:

• $h_{t-1}$: previous hidden state, a vector of dimension $h$
• $x_t$: current input, a vector of dimension $d$
• $[h_{t-1}, x_t]$: concatenation of the two vectors, producing a vector of dimension $d + h$
• $W_z, W_r, W_h$: learnable weight matrices, each of shape $(h, d + h)$
• $b_z, b_r, b_h$: learnable bias vectors, each of dimension $h$
• $\sigma$: the sigmoid function, $\sigma(x) = \frac{1}{1 + e^{-x}}$, outputs in $[0, 1]$
• $\tanh$: the hyperbolic tangent function, outputs in $[-1, 1]$
• $\odot$: element-wise (Hadamard) multiplication

The elegance of the GRU emerges from how these equations interact. Consider two extreme scenarios:

Scenario 1: Encountering a sentence boundary. When the model sees a period followed by a capital letter, it might learn to set the reset gate low (near 0) and the update gate high (near 1). The low reset gate means the candidate $\tilde{h}_t$ is computed almost entirely from the new input, ignoring the previous sentence's context. The high update gate then replaces most of the old hidden state with this fresh candidate. Result: the model "forgets" the previous sentence and starts fresh.

Scenario 2: Processing a long noun phrase. When the model is in the middle of "the large, spotted, energetic dog," it might learn to keep the reset gate high (preserving context about "the large, spotted") while setting the update gate to moderate values (blending new adjectives with existing understanding). Result: the model accumulates information about the noun phrase without losing track of what came before.

The network learns these patterns automatically through backpropagation, adjusting the weight matrices $W_z$, $W_r$, and $W_h$ to produce appropriate gate values for each situation it encounters during training.