Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GRU Architecture › GRU Parameter Efficiency

Michael BrenndoerferMay 14, 20254 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 80GRU Architecture › GRU Parameter Efficiency

Reading section 7 of 15 · GRU Parameter Efficiency · 4 minSection 7 of 15 · 4 min

Section 7 of 15 · 4 min read

GRU Architecture › GRU Parameter EfficiencyOne section · Sections in this chapter7 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of the GRU's main selling points is its reduced parameter count compared to LSTMs. Let's quantify this difference precisely.

For a single-layer recurrent network with input size $d$ and hidden size $h$ , each gate or transformation requires:

A weight matrix $W$ of shape $(h, d + h)$ , contributing $(d + h) \times h$ parameters
A bias vector $b$ of shape $(h,)$ , contributing $h$ parameters
Total per gate: $(d + h) \times h + h$ parameters

LSTM parameters (4 gates/transformations):

Forget gate: $(d + h) \times h + h$ parameters
Input gate: $(d + h) \times h + h$ parameters
Cell candidate: $(d + h) \times h + h$ parameters
Output gate: $(d + h) \times h + h$ parameters
Total: $4 \times [(d + h) \times h + h] = 4h^2 + 4dh + 4h$

GRU parameters (3 gates/transformations):

Reset gate: $(d + h) \times h + h$ parameters
Update gate: $(d + h) \times h + h$ parameters
Candidate: $(d + h) \times h + h$ parameters
Total: $3 \times [(d + h) \times h + h] = 3h^2 + 3dh + 3h$

The ratio of GRU to LSTM parameters is:

\frac{\text{GRU params}}{\text{LSTM params}} = \frac{3h^2 + 3dh + 3h}{4h^2 + 4dh + 4h} = \frac{3(h^2 + dh + h)}{4(h^2 + dh + h)} = \frac{3}{4} = 75\%

The GRU has exactly 75% of the LSTM's parameters, regardless of the input and hidden dimensions. For typical configurations, this translates to meaningful savings in memory, training time, and inference speed.

In[11]:

Code

def count_parameters(input_size, hidden_size, architecture):
    """Count parameters for LSTM or GRU."""
    d, h = input_size, hidden_size

    if architecture == "lstm":
        # 4 gates: forget, input, cell, output
        return 4 * ((d + h) * h + h)
    else:  # gru
        # 3 gates: reset, update, candidate
        return 3 * ((d + h) * h + h)


# Compare parameter counts for common configurations
configs = [
    (128, 256),  # Small model
    (256, 512),  # Medium model
    (512, 1024),  # Large model
]

print("Configuration     | LSTM Params | GRU Params | GRU/LSTM Ratio")
print("-" * 65)
for input_size, hidden_size in configs:
    lstm_params = count_parameters(input_size, hidden_size, "lstm")
    gru_params = count_parameters(input_size, hidden_size, "gru")
    ratio = gru_params / lstm_params
    print(
        f"d={input_size:3}, h={hidden_size:4} | {lstm_params:>11,} | {gru_params:>10,} | {ratio:.2%}"
    )

Out[12]:

Console

Configuration     | LSTM Params | GRU Params | GRU/LSTM Ratio
-----------------------------------------------------------------
d=128, h= 256 |     394,240 |    295,680 | 75.00%
d=256, h= 512 |   1,574,912 |  1,181,184 | 75.00%
d=512, h=1024 |   6,295,552 |  4,721,664 | 75.00%

The ratio remains exactly 75% regardless of the input or hidden dimensions, confirming our mathematical derivation. As models scale up, the absolute parameter savings become substantial: for a large model with hidden size 1024, the GRU saves over 3 million parameters compared to an equivalent LSTM. These savings translate directly to reduced memory usage during training and inference, faster gradient computations, and potentially better generalization on smaller datasets.

The 25% parameter reduction has several practical implications. Training is faster because there are fewer gradients to compute and fewer weights to update. Memory usage is lower, allowing larger batch sizes or longer sequences. And with fewer parameters, GRUs may be less prone to overfitting on smaller datasets.

The following table shows parameter counts for various hidden sizes with a fixed input size of 256:

Parameter count comparison between LSTM and GRU architectures. The GRU consistently uses exactly 75% of the LSTM's parameters regardless of hidden size, with absolute savings growing substantially as models scale.
Hidden Size	LSTM Parameters	GRU Parameters	GRU/LSTM Ratio
128	394,752	296,064	75%
256	1,050,624	787,968	75%
512	3,149,824	2,362,368	75%
768	6,300,672	4,725,504	75%
1024	10,489,856	7,867,392	75%