Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GRU Architecture › The Update Gate: Balancing Old and New

Michael BrenndoerferMay 14, 20257 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 80GRU Architecture › The Update Gate: Balancing Old and New

Reading section 4 of 15 · The Update Gate: Balancing Old and New · 7 minSection 4 of 15 · 7 min

Section 4 of 15 · 7 min read

GRU Architecture › The Update Gate: Balancing Old and NewOne section · Sections in this chapter4 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The update gate is the GRU's most distinctive feature. It performs double duty, simultaneously deciding how much of the old hidden state to retain and how much of the new candidate to incorporate. This elegant mechanism replaces the separate forget and input gates of the LSTM.

Out[7]:

Visualization

Diagram showing update gate creating weighted combination of old hidden state and new candidate. — The update gate creates a weighted average between the previous hidden state and the new candidate. A value of $z_t = 0.2$ means 'keep 80% old, add 20% new,' while $z_t = 0.8$ means 'keep 20% old, add 80% new.' This single gate replaces both the forget and input gates of LSTMs.

The update gate computation mirrors the reset gate's structure, using its own learned weights:

z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z)

where:

$z_t$ : the update gate output, a vector of dimension $h$ with values in $[0, 1]$
$W_z$ : the update gate's weight matrix of shape $(h, d + h)$
$[h_{t-1}, x_t]$ : concatenation of previous hidden state and current input
$b_z$ : the update gate's bias vector of dimension $h$
$\sigma$ : the sigmoid activation function

The key insight is how $z_t$ is used in the final state update. Rather than simply replacing the old state with the new candidate, the GRU computes a weighted average:

h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t

where:

$h_t$ : the new hidden state at timestep $t$
$h_{t-1}$ : the previous hidden state from timestep $t-1$
$\tilde{h}_t$ : the candidate hidden state (computed using the reset gate)
$z_t$ : the update gate output, controlling the interpolation
$\odot$ : element-wise (Hadamard) multiplication
$(1 - z_t)$ : the complement of the update gate, representing "how much to keep"

This formula creates a smooth interpolation between the old state and the new candidate. Consider what happens at the extremes:

When $z_t = 0$ : the formula becomes $h_t = 1 \cdot h_{t-1} + 0 \cdot \tilde{h}_t = h_{t-1}$ . The hidden state is copied directly from the previous timestep with no modification, creating a perfect information highway.
When $z_t = 1$ : the formula becomes $h_t = 0 \cdot h_{t-1} + 1 \cdot \tilde{h}_t = \tilde{h}_t$ . The hidden state is completely replaced by the new candidate.
When $z_t = 0.3$ : the formula becomes $h_t = 0.7 \cdot h_{t-1} + 0.3 \cdot \tilde{h}_t$ . The new state is 70% old information and 30% new.

This weighted average is computed element-wise, so different dimensions of the hidden state can have different update rates. Some dimensions might preserve information across many timesteps while others update frequently.

Out[8]:

Visualization

Line plot showing how the hidden state interpolates between old and new values as the update gate varies from 0 to 1. — The update gate creates a smooth interpolation between the previous hidden state and the new candidate. As $z_t$ increases from 0 to 1, the output transitions from purely old information (blue) to purely new information (green). The intermediate values create weighted blends.

Update Gate vs Forget + Input Gates

In LSTMs, the forget gate and input gate operate independently. You could theoretically forget everything while adding nothing, or add new information without forgetting anything. The GRU's update gate enforces a constraint: the amount you forget and the amount you add must sum to 1. This constraint reduces flexibility but also reduces the parameter count and can act as a regularizer.