Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

FFN Activation Functions › ReLU: The Original Choice

Michael BrenndoerferUpdated June 14, 20259 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 108FFN Activation Functions › ReLU: The Original Choice

Reading section 3 of 15 · ReLU: The Original Choice · 9 minSection 3 of 15 · 9 min

Section 3 of 15 · 9 min read

FFN Activation Functions › ReLU: The Original ChoiceOne section · Sections in this chapter3 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The Rectified Linear Unit (ReLU) is the simplest nonlinear activation function in modern deep learning. It applies a trivial rule: keep positive values unchanged, set negative values to zero. Mathematically:

\text{ReLU}(x) = \max(0, x)

where:

$x$ : the input value (a single element of the pre-activation vector)
$\max(0, x)$ : returns $x$ if $x > 0$ , otherwise returns $0$

This simplicity is deceptive. ReLU revolutionized deep learning when introduced in AlexNet (2012), enabling training of much deeper networks than the sigmoid and tanh activations that preceded it. The original transformer adopted ReLU for its feed-forward layers, inheriting this proven workhorse.

Think of ReLU as a one-way valve. Values trying to flow in the positive direction pass through without resistance. Values trying to flow in the negative direction are blocked completely. The network learns to use this valve to select which "features" in the hidden representation are worth propagating and which should be silenced.

The key insight behind ReLU's success is its effect on gradients. Sigmoid and tanh both saturate: for very large or very small inputs, their derivatives approach zero, making it nearly impossible for deep layers to receive useful gradient signal. ReLU does not saturate for positive inputs. Its derivative is exactly 1 for any positive value, meaning gradient flows through positive activations without shrinkage. This simple property allowed researchers to train networks twenty, fifty, even one hundred layers deep, something that had been essentially impossible with saturating activations.

Rectified Linear Unit (ReLU)

A piecewise linear activation function that outputs the input directly if positive, otherwise outputs zero. Its simplicity enables fast computation and its non-saturating positive region prevents vanishing gradients during backpropagation.

Let's implement ReLU and examine its behavior:

In[3]:

Code

import numpy as np

np.random.seed(42)


def relu(x):
    """Rectified Linear Unit activation."""
    return np.maximum(0, x)


def relu_derivative(x):
    """Derivative of ReLU."""
    return (x > 0).astype(float)


# Generate input range
x = np.linspace(-3, 3, 1000)

Out[4]:

Visualization

Plot showing ReLU function as a bent line at the origin. — ReLU activation function across the input range [-3, 3]. Negative inputs are zeroed out entirely, while positive inputs pass through unchanged. The sharp corner at the origin is the defining characteristic of ReLU and the source of both its efficiency and its limitations.

Plot showing ReLU derivative as a step function. — ReLU derivative. The derivative is a step function: 0 for negative inputs, 1 for positive inputs. The sharp corner at zero is both ReLU''s strength (computational simplicity) and weakness (non-smooth gradients).

ReLU's advantages are clear. Its computation is trivial: a single comparison operation. The derivative is equally simple:

\text{ReLU}'(x) = \begin{cases} 1 & \text{if } x > 0 \\ 0 & \text{if } x \leq 0 \end{cases}

where:

The derivative is 1 for positive inputs, meaning gradients flow through unchanged
The derivative is 0 for negative inputs, meaning gradients are blocked entirely
The derivative is technically undefined at exactly $x = 0$ , but implementations typically use 0 or 1

This step function gradient never vanishes for positive inputs like sigmoid's gradient does for large values. And ReLU creates sparse activations, as roughly half of all hidden units output zero for typical input distributions, which can improve efficiency and interpretability.

In practice, the sparsity ReLU creates is both a feature and a design principle. Sparse representations are more interpretable: when only a subset of neurons activate for a given input, you can potentially trace which features are "firing" for which inputs. Sparse activations also enable optimizations in hardware, since multiplying by zero can be skipped. In very large FFN layers (some models use $d_{\text{ff}} = 16{,}384$ or larger), exploiting this sparsity can reduce compute.

The Dead ReLU ProblemLink Copied

But ReLU has a critical weakness. When a neuron's pre-activation is consistently negative, its gradient is always zero. The neuron stops learning entirely. This "dying ReLU" problem becomes more severe as networks deepen or learning rates increase.

Consider what happens during training. If a weight update pushes a neuron's bias too negative, that neuron's output becomes zero for all inputs. With zero output, the gradient flowing through that neuron is zero. With zero gradient, the weights never update. The neuron is dead.

The cascading nature of this problem makes it particularly insidious. A dead neuron in an early layer of the FFN prevents any downstream computation from receiving information through that pathway. If many neurons die, the effective width of the network shrinks, reducing its capacity. In extreme cases, if a whole layer becomes largely dead, the model trains without using that layer at all, wasting parameters and compute.

Dead neurons arise most commonly from two causes. The first is a high initial learning rate: a large gradient step early in training can push many neurons into the negative half-plane simultaneously, and if the learning rate remains high, the neurons never recover. The second is weight initialization that creates very negative initial biases, giving neurons little chance of activating before training has even begun. Careful initialization (like He initialization, which accounts for ReLU's expected output variance) reduces the risk but does not eliminate it.

In[5]:

Code

# Simulate dying ReLU phenomenon
def simulate_dead_neurons(d_ff, n_samples, bias_shift=-2.0):
    """
    Simulate FFN hidden layer activations with shifted bias.

    Returns the fraction of neurons that are 'dead' (always zero).
    """
    # Random input activations (post-attention representations)
    inputs = np.random.randn(n_samples, d_ff)

    # Simulate pre-activations with negative bias shift
    pre_activations = inputs + bias_shift

    # Apply ReLU
    activations = relu(pre_activations)

    # Count neurons that are zero for all samples
    neuron_max = activations.max(axis=0)
    dead_fraction = (neuron_max == 0).mean()

    return dead_fraction, activations


# Test different bias shifts
bias_shifts = np.linspace(0, -3, 7)
dead_fractions = []

for shift in bias_shifts:
    # A modest batch exposes how negative bias can silence a neuron across all
    # examples, while keeping the comparison deterministic across render themes.
    dead_frac, _ = simulate_dead_neurons(1000, 64, shift)
    dead_fractions.append(dead_frac)

Out[6]:

Visualization

Line plot showing dead neuron fraction increasing from 0% at bias shift 0 to more than 90% at bias shift -3. — Dead neuron fraction as a function of bias shift in a simulated 1000-neuron ReLU layer evaluated over a batch of 64 inputs. As biases become more negative (x-axis moves right), an increasing fraction of neurons output zero for every input in the batch, effectively disconnecting those neurons from the computation graph. At a bias shift of -3.0, more than 90% of neurons are dead and the layer has lost most of its representational capacity.

The plot shows how quickly neurons die as bias values shift negative. In real training, this can happen in patches of the network, reducing effective capacity without obvious symptoms. The model trains, but parts of it have effectively disconnected from the computation.

Notice that the dying-neuron problem has no easy diagnostic signal during training. The loss might still decrease, the validation metrics might still improve, but you could be doing so with only a fraction of the network's intended capacity. This silent degradation motivated researchers to search for activations that provide nonzero gradients everywhere, even for negative inputs.

ReLU Variants: Partial FixesLink Copied

Several ReLU variants attempted to address the dead neuron problem before GELU and SiLU became the dominant alternatives. Leaky ReLU uses a small negative slope (typically 0.01) for negative inputs instead of zeroing them:

\text{LeakyReLU}(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha x & \text{if } x \leq 0 \end{cases}

where:

$\alpha$ : a small positive constant (typically 0.01), sometimes called the "leak" coefficient

By allowing a tiny gradient for negative inputs, Leaky ReLU prevents neurons from dying completely. Parametric ReLU (PReLU) takes this further by making $\alpha$ a learnable parameter, allowing the network to decide how much of the negative signal to preserve. ELU (Exponential Linear Unit) uses a smooth exponential function for negative inputs, avoiding the discontinuity entirely.

These variants were steps in the right direction, but none of them fully addressed the deeper issue: ReLU's sharp corner at zero creates an abrupt transition that can destabilize training in large models. Researchers ultimately needed a non-zero negative gradient inside a smooth, differentiable function throughout its entire domain. That insight led to GELU.