Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

FFN Activation Functions › Computational Efficiency

Michael BrenndoerferUpdated June 14, 20253 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 108FFN Activation Functions › Computational Efficiency

Reading section 9 of 15 · Computational Efficiency · 3 minSection 9 of 15 · 3 min

Section 9 of 15 · 3 min read

FFN Activation Functions › Computational EfficiencyOne section · Sections in this chapter9 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Activation function speed matters at scale. When processing billions of tokens through trillion-parameter models, even small differences in activation computation time accumulate. The helper below shows how to benchmark the implementations. For the visualization, we use one fixed reference measurement so the data remains identical across light, dark, and transparent render variants:

In[23]:

Code

import time


def benchmark_activation(func, x, n_iterations=100):
    """Benchmark activation function speed."""
    # Warmup
    for _ in range(10):
        _ = func(x)

    # Timed runs
    start = time.perf_counter()
    for _ in range(n_iterations):
        _ = func(x)
    elapsed = time.perf_counter() - start

    return elapsed / n_iterations * 1000  # ms per call


# Representative ratios from one NumPy/SciPy CPU reference run. Keeping this
# measurement fixed prevents theme-rendering overhead from changing the bars.
benchmarks = {
    "ReLU": 1.0,
    "GELU (exact)": 32.0,
    "GELU (approx)": 29.5,
    "SiLU": 6.8,
}

Out[24]:

Visualization

Horizontal bar chart showing relative computation times with ReLU fastest at 1x and exact GELU slowest. — Representative NumPy/SciPy CPU timing ratios for activation functions on a 1024x4096 tensor, normalized relative to ReLU. ReLU is fastest because it requires only a comparison operation. In this reference measurement, SiLU is faster than both GELU variants and exact GELU using SciPy's error function is slowest. Exact ratios depend on the software and hardware; dedicated GPU implementations commonly use fused kernels and can differ substantially.

In this reference measurement, ReLU is fastest because it is just a comparison operation. The exact GELU using erf is slowest. The GELU tanh approximation is faster than exact GELU but still slower than SiLU. SiLU is relatively fast because sigmoid has efficient implementations on most hardware.

In practice, these differences are often dwarfed by memory bandwidth limitations and matrix multiplication costs. The activation function typically accounts for less than 1% of total FFN compute time. Still, at sufficient scale, even small improvements matter.

This reference measurement represents NumPy/SciPy CPU performance and does not reflect production GPU performance. On modern GPUs, activation functions are often "fused" into preceding or following operations using custom CUDA kernels. A fused kernel computes the matrix multiplication and activation in a single pass over memory, reducing the number of times data must be read from GPU memory. With fused operations, the relative cost of more expensive activations (like exact GELU) shrinks significantly, because the bottleneck shifts from computation to memory bandwidth. This is one reason why the exact GELU is feasible in production despite its higher computational cost.