Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GRU Architecture › Training Time Comparison

Michael BrenndoerferMay 14, 20252 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 80GRU Architecture › Training Time Comparison

Reading section 11 of 15 · Training Time Comparison · 2 minSection 11 of 15 · 2 min

Section 11 of 15 · 2 min read

GRU Architecture › Training Time ComparisonOne section · Sections in this chapter11 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's measure the actual training speed difference between GRUs and LSTMs.

In[22]:

Code

import time


def measure_training_time(
    model_class, input_size, hidden_size, seq_length, num_batches=100
):
    """Measure training time for a model."""
    model = model_class(input_size, hidden_size, input_size)
    optimizer = torch.optim.Adam(model.parameters())
    criterion = nn.CrossEntropyLoss()

    # Warm up
    x = torch.randn(32, seq_length, input_size)
    labels = torch.randint(0, input_size, (32,))
    for _ in range(10):
        optimizer.zero_grad()
        outputs = model(x)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

    # Timed run
    start = time.time()
    for _ in range(num_batches):
        x = torch.randn(32, seq_length, input_size)
        labels = torch.randint(0, input_size, (32,))
        optimizer.zero_grad()
        outputs = model(x)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

    return time.time() - start


# Compare training times
configs = [(64, 100), (128, 200), (256, 300)]
gru_times = []
lstm_times = []

for hidden_size, seq_length in configs:
    gru_time = measure_training_time(GRUClassifier, 8, hidden_size, seq_length)
    lstm_time = measure_training_time(
        LSTMClassifier, 8, hidden_size, seq_length
    )
    gru_times.append(gru_time)
    lstm_times.append(lstm_time)

Out[23]:

Console

Configuration          | GRU Time | LSTM Time | Speedup
------------------------------------------------------------
h= 64, seq_len=100 |    1.07s |     1.20s | 1.13x
h=128, seq_len=200 |    4.47s |     5.52s | 1.24x
h=256, seq_len=300 |    9.08s |    12.32s | 1.36x

The GRU consistently trains faster than the LSTM across all configurations tested:

Training time comparison for 100 batches across different model configurations. The speedup reflects the 25% parameter reduction, with larger models showing more pronounced savings.
Configuration	GRU Time	LSTM Time	GRU Speedup
h=64, seq=100	~2.5s	~2.8s	~1.12×
h=128, seq=200	~5.5s	~6.5s	~1.18×
h=256, seq=300	~12s	~15s	~1.25×

The speedup reflects the 25% reduction in parameters: fewer weights mean fewer gradient computations during backpropagation. The speedup tends to be more pronounced for larger models and longer sequences, where the computational savings accumulate. While a 1.1-1.3× speedup might seem modest for a single training run, it compounds significantly during hyperparameter searches, cross-validation, or when training multiple models. For a project requiring 100 training runs, this difference could save hours or even days of compute time.