Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

FFN Activation Functions › Activation Function Comparison

Michael BrenndoerferUpdated June 14, 20255 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 108FFN Activation Functions › Activation Function Comparison

Reading section 6 of 15 · Activation Function Comparison · 5 minSection 6 of 15 · 5 min

Section 6 of 15 · 5 min read

FFN Activation Functions › Activation Function ComparisonOne section · Sections in this chapter6 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

With all three activations implemented, let's visualize them together to understand their relationships and differences:

Out[14]:

Visualization

Plot showing three activation functions overlaid with ReLU as a bent line and GELU and SiLU as smooth curves with negative dips. — Comparison of ReLU, GELU, and SiLU activation functions across [-3, 3]. All three share the property of being approximately linear for large positive inputs and approximately zero for large negative inputs. The critical differences lie in the transition region near zero: ReLU has a sharp corner and no negative outputs, while GELU and SiLU are smooth with slight negative dips that provide gradient signal even for mildly negative pre-activations.

The three functions converge for large positive inputs, approaching the identity function. They also converge for large negative inputs, approaching zero. The critical differences are in the transition region around zero, where:

ReLU has a sharp corner with discontinuous derivative
GELU has a smooth transition with a slight negative dip (minimum $\approx -0.17$ at $x \approx -0.5$ )
SiLU has a smooth transition with a deeper negative dip (minimum $\approx -0.28$ at $x \approx -1.28$ )

Let's also compare the derivatives, which directly affect gradient flow during training:

Out[15]:

Visualization

Plot showing three derivative curves with ReLU as a step function and GELU and SiLU as smooth sigmoid-like curves. — Derivatives of ReLU, GELU, and SiLU across [-3, 3]. ReLU''s derivative is a discontinuous step function that is either 0 or 1. GELU and SiLU have smooth derivatives that transition gradually from near 0 to near 1. Both smooth functions mildly overshoot 1 for positive inputs: GELU peaks earlier and slightly higher, while SiLU peaks later and lower.

The derivative plots show that both smooth activations can mildly amplify gradients for certain positive inputs. GELU's derivative peaks at approximately $1.129$ near $x = 1.41$ , while SiLU's derivative peaks at approximately $1.100$ near $x = 2.40$ . Let's examine SiLU's broader amplification region in more detail:

Out[16]:

Visualization

Line plot of SiLU derivative with shaded region showing where derivative exceeds 1.0. — Gradient amplification region for SiLU across [-2, 4]. The shaded region shows where the SiLU derivative exceeds 1.0, meaning gradients are mildly amplified rather than attenuated during backpropagation. The derivative crosses 1 near x = 1.28, peaks near x = 2.40, and remains just above 1 through the displayed range while tapering back toward 1.

Quantitative ComparisonLink Copied

Let's compute specific properties that affect neural network behavior:

In[17]:

Code

def analyze_activation(name, func, x_range):
    """Compute key properties of an activation function."""
    y = func(x_range)

    # Find minimum value and location
    min_idx = np.argmin(y)
    min_val = y[min_idx]
    min_x = x_range[min_idx]

    # Sparsity: fraction of outputs that are zero or near-zero
    sparsity = (np.abs(y) < 0.01).mean()

    # Linearity for positive inputs (correlation with y=x)
    pos_mask = x_range > 0.5
    correlation = np.corrcoef(x_range[pos_mask], y[pos_mask])[0, 1]

    return {
        "name": name,
        "min_value": min_val,
        "min_location": min_x,
        "sparsity": sparsity,
        "linearity": correlation,
    }


x_analysis = np.linspace(-5, 5, 10000)
activations = [
    ("ReLU", relu),
    ("GELU", gelu_exact),
    ("SiLU", silu),
]

results = [
    analyze_activation(name, func, x_analysis) for name, func in activations
]

Out[18]:

Console

Activation function properties:

Function      Min Value   Min Location     Sparsity    Linearity
-----------------------------------------------------------------
ReLU             0.0000          -5.00       50.1%     1.000000
GELU            -0.1700          -0.75       23.6%     0.999769
SiLU            -0.2785          -1.28        0.4%     0.999823

Key observations from this analysis:

Minimum value: ReLU never goes negative (by definition). GELU's dip is mild (-0.17), while SiLU dips deeper (-0.28). These negative values can help with regularization but also introduce potential instabilities.
Sparsity: All three create some degree of sparsity (outputs near zero), but ReLU is the most sparse for normally distributed inputs. This sparsity can aid interpretability and efficiency.
Linearity: All three are highly linear for positive inputs (correlation > 0.999 with $y = x$ ), which helps preserve signal magnitude through deep networks.

The sparsity difference between ReLU and the smooth activations is worth dwelling on. For a standard normal input distribution, ReLU produces exactly 50% sparsity by zeroing all negative values. GELU and SiLU produce much less sparsity because they only attenuate negative inputs, not zero them. This means GELU and SiLU FFN layers operate with denser intermediate representations, which requires slightly more memory bandwidth during training and inference.