Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GRU Architecture › Implementing a GRU from Scratch

Michael BrenndoerferMay 14, 202510 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 80GRU Architecture › Implementing a GRU from Scratch

Reading section 8 of 15 · Implementing a GRU from Scratch · 10 minSection 8 of 15 · 10 min

Section 8 of 15 · 10 min read

GRU Architecture › Implementing a GRU from ScratchOne section · Sections in this chapter8 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Theory becomes concrete through implementation. By building a GRU cell from scratch using only NumPy, we'll see exactly how the mathematical equations translate into executable code. This exercise reveals the simplicity underlying the architecture: despite the sophisticated behavior GRUs can learn, the forward pass is just a sequence of matrix multiplications, element-wise operations, and activation functions.

Setting Up the Activation FunctionsLink Copied

Before implementing the GRU itself, we need the two activation functions that appear in the equations. The sigmoid function $\sigma$ maps any real number to the interval $[0, 1]$ , making it perfect for gates. The tanh function maps to $[-1, 1]$ , centering values around zero for the candidate computation.

In[13]:

Code

import numpy as np


def sigmoid(x):
    """Numerically stable sigmoid function."""
    return np.where(x >= 0, 1 / (1 + np.exp(-x)), np.exp(x) / (1 + np.exp(x)))


def tanh(x):
    """Hyperbolic tangent function."""
    return np.tanh(x)

The sigmoid implementation uses a numerically stable form that avoids overflow for large negative inputs. For $x \geq 0$ , we compute $\frac{1}{1 + e^{-x}}$ directly. For $x < 0$ , we use the equivalent form $\frac{e^x}{1 + e^x}$ , which avoids computing $e^{-x}$ for large negative $x$ .

The GRU Cell ClassLink Copied

Now we can implement the GRU cell. The class needs to store three sets of parameters (one for each gate/transformation) and implement the forward pass that computes the four steps we discussed.

In[14]:

Code

class GRUCell:
    """A single GRU cell implemented from scratch."""

    def __init__(self, input_size, hidden_size):
        self.input_size = input_size
        self.hidden_size = hidden_size

        # Initialize weights using Xavier initialization
        scale = np.sqrt(2.0 / (input_size + hidden_size))

        # Reset gate parameters
        self.W_r = (
            np.random.randn(hidden_size, input_size + hidden_size) * scale
        )
        self.b_r = np.zeros((hidden_size, 1))

        # Update gate parameters
        self.W_z = (
            np.random.randn(hidden_size, input_size + hidden_size) * scale
        )
        self.b_z = np.zeros((hidden_size, 1))

        # Candidate hidden state parameters
        self.W_h = (
            np.random.randn(hidden_size, input_size + hidden_size) * scale
        )
        self.b_h = np.zeros((hidden_size, 1))

    def forward(self, x, h_prev):
        """
        Forward pass through the GRU cell.

        Args:
            x: Input at current timestep, shape (input_size, 1)
            h_prev: Previous hidden state, shape (hidden_size, 1)

        Returns:
            h_new: New hidden state, shape (hidden_size, 1)
        """
        # Concatenate input and previous hidden state
        concat = np.vstack([h_prev, x])

        # Step 1: Compute reset gate
        r = sigmoid(self.W_r @ concat + self.b_r)

        # Step 2: Compute update gate
        z = sigmoid(self.W_z @ concat + self.b_z)

        # Step 3: Compute candidate hidden state
        # Note: We concatenate (r * h_prev) with x
        reset_hidden = r * h_prev
        concat_reset = np.vstack([reset_hidden, x])
        h_candidate = tanh(self.W_h @ concat_reset + self.b_h)

        # Step 4: Compute final hidden state
        h_new = (1 - z) * h_prev + z * h_candidate

        return h_new, {"r": r, "z": z, "h_candidate": h_candidate}

Let's trace through the key parts of this implementation:

Weight initialization: We use Xavier initialization, scaling weights by $\sqrt{\frac{2}{d + h}}$ where $d$ is the input dimension and $h$ is the hidden dimension. This keeps the variance of activations roughly constant across layers, which helps with training stability.
Concatenation: The line concat = np.vstack([h_prev, x]) creates the $[h_{t-1}, x_t]$ vector that both gates receive as input. By stacking vertically, we create a single vector of dimension $d + h$ .
Gate computations: Each gate follows the same pattern: matrix multiply, add bias, apply sigmoid. The @ operator performs matrix multiplication in NumPy.
Reset-gated candidate: The candidate computation is slightly more complex because it uses the reset-gated hidden state. We first compute reset_hidden = r * h_prev (element-wise multiplication), then concatenate this with the input before the matrix multiplication.
Final blending: The last line implements the weighted average formula, using NumPy's broadcasting for element-wise operations.

Testing the ImplementationLink Copied

Let's verify our implementation by processing a short sequence and examining how the gates behave. With randomly initialized weights, we expect the gates to output values near 0.5 (since sigmoid of values near zero is approximately 0.5).

In[15]:

Code

# Create a GRU cell
gru = GRUCell(input_size=4, hidden_size=8)

# Create a sample sequence (3 timesteps, 4 features each)
sequence = np.random.randn(3, 4, 1)

# Initialize hidden state
h = np.zeros((8, 1))

# Process the sequence
hidden_states = []
gate_activations = []

for t in range(len(sequence)):
    h, gates = gru.forward(sequence[t], h)
    hidden_states.append(h.copy())
    gate_activations.append(gates)

Out[16]:

Console

Processing sequence of length 3:
--------------------------------------------------

Timestep 0:
  Reset gate mean:  0.517
  Update gate mean: 0.457
  Hidden state L2:  0.426

Timestep 1:
  Reset gate mean:  0.530
  Update gate mean: 0.466
  Hidden state L2:  0.578

Timestep 2:
  Reset gate mean:  0.470
  Update gate mean: 0.542
  Hidden state L2:  0.969

The results confirm our expectations. With randomly initialized weights, the gate activations hover around 0.5. This reflects the sigmoid function's output for inputs near zero. The reset gate mean of approximately 0.5 indicates that the network is partially incorporating past context into the candidate computation. Similarly, the update gate mean near 0.5 means the hidden state update is roughly half old information and half new candidate.

Notice how the hidden state L2 norm grows modestly across timesteps. Starting from zero, the hidden state accumulates information as each timestep contributes new content. The growth is bounded because tanh keeps the candidate values in $[-1, 1]$ , and the update gate's blending prevents explosive growth.

Out[17]:

Visualization

Line plot showing hidden state L2 norm growth over 30 timesteps. — The hidden state L2 norm grows initially as information accumulates from the input sequence, then stabilizes due to tanh bounding and update gate averaging.

Multi-line plot showing 8 hidden state dimensions evolving over timesteps. — Individual dimensions follow diverse trajectories, allowing the GRU to track multiple aspects of the input simultaneously.

The left panel shows that the hidden state's magnitude (L2 norm) grows initially as information from the input sequence accumulates, then tends to stabilize. This bounded growth is a direct consequence of the tanh activation, which constrains candidate values to $[-1, 1]$ , and the update gate's weighted averaging, which prevents any single timestep from causing explosive changes.

The right panel reveals that different dimensions of the hidden state follow diverse trajectories. Some dimensions oscillate around zero, others trend positive or negative, and some remain relatively stable. This diversity is essential: it allows the GRU to track multiple aspects of the input sequence simultaneously. After training, these dimensions would specialize to capture different semantic features relevant to the task.

After training on actual data, these gate activations would show more pronounced patterns. Instead of hovering near 0.5, they would swing closer to 0 or 1 depending on the learned task requirements. A language model, for instance, might learn to push the reset gate toward 0 at sentence boundaries and toward 1 within sentences.

Let's visualize how the gates and hidden state evolve across a longer sequence to see the full picture of GRU dynamics.

Out[18]:

Visualization

Heatmap of reset gate values across 20 timesteps and 8 dimensions, colored from blue to red. — Reset gate activations across a 20-timestep sequence for 8 hidden dimensions. Values near 0 (blue) suppress past state influence; values near 1 (red) incorporate full past context.

Heatmap of update gate values across 20 timesteps and 8 dimensions, colored from blue to red. — Update gate activations showing how much each dimension updates per timestep. Low values copy the old hidden state; high values replace it with the new candidate.

Heatmap of hidden state values across 20 timesteps and 8 dimensions, colored from blue to red. — Hidden state values bounded to the tanh range of [-1, 1]. Diverse trajectories across dimensions show each specializes to track different input features.

The heatmaps reveal several key insights about GRU behavior. First, notice how different dimensions of the hidden state can have very different activation patterns: some dimensions maintain relatively stable values while others fluctuate more. Second, the gate values cluster around 0.5 with random weights, but after training, we would expect to see more extreme values (closer to 0 or 1) at positions where the network has learned to make clear decisions about information flow. Third, the hidden state values remain bounded within $[-1, 1]$ due to the tanh activation in the candidate computation, preventing runaway activations.