Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

FFN Activation Functions › GELU: The Smooth Alternative

Michael BrenndoerferUpdated June 14, 202510 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 108FFN Activation Functions › GELU: The Smooth Alternative

Reading section 4 of 15 · GELU: The Smooth Alternative · 10 minSection 4 of 15 · 10 min

Section 4 of 15 · 10 min read

FFN Activation Functions › GELU: The Smooth AlternativeOne section · Sections in this chapter4 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Hendrycks and Gimpel introduced the Gaussian Error Linear Unit (GELU) in 2016, though it didn't gain widespread adoption until BERT popularized it in 2018. GELU addresses ReLU's sharp corner by introducing smooth, probabilistic gating.

The intuition behind GELU is elegant: instead of deterministically zeroing negative values, gate each input by its probability of being positive under a standard normal distribution. Inputs that are clearly positive (large positive values) pass through nearly unchanged. Inputs that are clearly negative (large negative values) are nearly zeroed. Inputs near zero, where classification is uncertain, are partially attenuated.

Think of GELU as a soft decision boundary instead of a hard one. Whereas ReLU says "if you're negative, you're blocked, full stop," GELU says "if you're probably negative, you're mostly blocked, but we'll keep a little signal proportional to how uncertain we are." This probabilistic framing connects activation functions to Bayesian reasoning: the gate is literally a probability estimate.

The mathematical formulation follows this intuition:

\text{GELU}(x) = x \cdot \Phi(x)

where:

$x$ : the input value to the activation function
$\Phi(x)$ : the cumulative distribution function (CDF) of the standard normal distribution, representing the probability that a standard normal random variable is less than or equal to $x$

The CDF itself is computed using the error function:

\Phi(x) = P(X \leq x) = \frac{1}{2}\left[1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right]

where:

$P(X \leq x)$ : the probability that a standard normal random variable $X$ takes a value less than or equal to $x$
$\text{erf}(\cdot)$ : the Gauss error function, a special function that arises in probability and statistics
$\sqrt{2}$ : a scaling factor that converts from the standard normal to the error function's parameterization

The CDF $\Phi(x)$ ranges from 0 to 1. For large positive $x$ , $\Phi(x) \approx 1$ , so $\text{GELU}(x) \approx x$ . For large negative $x$ , $\Phi(x) \approx 0$ , so $\text{GELU}(x) \approx 0$ . The transition is smooth, governed by the bell curve of the normal distribution.

The key insight here is the choice of the normal distribution as the gating mechanism. By using $\Phi(x)$ , we are essentially asking: "if I draw a random variable from a standard normal, how likely is it to be less than this input?" For a pre-activation value of 0, the answer is 50%, so GELU attenuates zero-valued inputs by half. For a pre-activation value of 2, the answer is about 97.7%, so GELU passes almost all of that signal. This creates a natural, data-independent scale for "how much" to pass each value.

Gaussian Error Linear Unit (GELU)

An activation function that gates inputs by their Gaussian CDF values, creating a smooth, non-monotonic function that approximates stochastic regularization. GELU is the standard activation for encoder-style transformers like BERT and RoBERTa.

In[7]:

Code

from scipy.special import erf


def gelu_exact(x):
    """GELU activation using exact computation."""
    return x * 0.5 * (1 + erf(x / np.sqrt(2)))


def gelu_approximate(x):
    """GELU approximation using tanh (faster, commonly used)."""
    return 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x**3)))


def gelu_derivative(x):
    """Derivative of GELU (exact)."""
    phi = 0.5 * (1 + erf(x / np.sqrt(2)))
    pdf = np.exp(-0.5 * x**2) / np.sqrt(2 * np.pi)
    return phi + x * pdf

GELU has a distinctive shape: it's nearly linear for positive values, smoothly transitions through zero, and has a slight dip into negative territory before approaching zero asymptotically. This dip is unique. For small negative inputs (around $x \approx -0.5$ ), GELU outputs slightly negative values. This non-monotonic behavior distinguishes it from ReLU variants.

The derivative of GELU, which determines gradient flow during backpropagation, is:

\text{GELU}'(x) = \Phi(x) + x \cdot \phi(x)

where:

$\Phi(x)$ : the standard normal CDF (as defined above)
$\phi(x) = \frac{1}{\sqrt{2\pi}} e^{-x^2/2}$ : the standard normal probability density function (PDF)
The first term $\Phi(x)$ contributes the "identity-like" gradient for positive inputs
The second term $x \cdot \phi(x)$ creates a smooth transition region and ensures the derivative is continuous everywhere

Notice that this derivative is never exactly zero except in the limit as $x \to -\infty$ . For any finite input, GELU provides some gradient signal, addressing the dying-neuron problem that afflicts ReLU. The gradient is small for very negative inputs, but it is nonzero. This means GELU neurons can, in principle, recover from temporarily negative pre-activations during training.

Out[8]:

Visualization

Plot showing GELU as a smooth S-shaped curve with slight negative dip. — GELU activation function across [-3, 3]. Unlike ReLU, GELU is smooth everywhere and non-monotonic, with a slight negative dip around x = -0.5. The dashed line shows the identity y = x, illustrating how GELU approaches linear behavior for large positive inputs.

Plot showing GELU derivative as a smooth sigmoid-like curve. — GELU derivative across [-3, 3]. The derivative transitions smoothly from near 0 for large negative inputs to near 1 for large positive inputs, avoiding ReLU's discontinuity at the origin and providing gradient signal across the full input range.

Worked Example: Tracing a Value Through GELULink Copied

To build concrete intuition, let's trace a single pre-activation value through the GELU computation step by step. Suppose the FFN's first layer produces a pre-activation of $x = 1.0$ for a particular neuron.

Step 1: Compute the CDF value at $x = 1.0$ . The standard normal CDF at 1.0 is $\Phi(1.0) \approx 0.8413$ . This says: "there is an 84.1% probability that a standard normal variable is at most 1.0."

Step 2: Multiply the input by this CDF value:

\text{GELU}(1.0) = 1.0 \times 0.8413 \approx 0.841

The neuron passes about 84% of its signal. Now suppose the pre-activation is $x = -1.0$ . The CDF value is $\Phi(-1.0) \approx 0.1587$ , so:

\text{GELU}(-1.0) = -1.0 \times 0.1587 \approx -0.159

The neuron passes about 16% of its (negative) signal, attenuating it substantially but not zeroing it. Compare this to ReLU, which would output 0 for $x = -1.0$ and 1.0 for $x = 1.0$ . GELU's output is softer in both cases: positive activations are attenuated slightly, negative activations are attenuated heavily but remain nonzero.

This soft gating behavior, controlled by the Gaussian CDF, is what makes GELU particularly well-suited to the pre-activation distributions that arise in transformer FFNs. Transformer pre-activations after LayerNorm are approximately normally distributed. GELU's gating function is calibrated to the same normal distribution, making the soft-gating particularly well-aligned with the actual statistical properties of the data it operates on.

GELU ApproximationsLink Copied

Computing the exact error function is expensive. In practice, transformers use fast approximations. The most common is the tanh approximation:

\text{GELU}_{\text{approx}}(x) = 0.5 \cdot x \cdot \left(1 + \tanh\left(\sqrt{\frac{2}{\pi}}\left(x + 0.044715 x^3\right)\right)\right)

where:

$x$ : the input value
$\tanh(\cdot)$ : the hyperbolic tangent function, which outputs values in the range $(-1, 1)$
$\sqrt{2/\pi} \approx 0.7979$ : a scaling constant derived from properties of the normal distribution
$0.044715$ : an empirically-fitted coefficient that improves the approximation accuracy
$x + 0.044715 x^3$ : a polynomial that approximates the argument to the error function

This formula looks complex, but it avoids the expensive erf computation while matching GELU closely. The constants were determined by fitting to the exact GELU curve, minimizing the maximum error across typical input ranges.

The reason this approximation works so well is that $\tanh$ and $\text{erf}$ are closely related functions. Both are odd functions (symmetric about the origin), both map the real line to $(-1, 1)$ , and both have an S-shaped curve. The polynomial $x + 0.044715 x^3$ inside the $\tanh$ call adjusts the rate of transition to closely match the CDF shape. The result is an approximation that differs from the exact GELU by at most about 0.004 across the full input range.

Let's compare the exact and approximate versions:

In[9]:

Code

# Compare exact and approximate GELU
x_test = np.linspace(-4, 4, 1000)
gelu_exact_vals = gelu_exact(x_test)
gelu_approx_vals = gelu_approximate(x_test)

max_error = np.max(np.abs(gelu_exact_vals - gelu_approx_vals))
mean_error = np.mean(np.abs(gelu_exact_vals - gelu_approx_vals))

Out[10]:

Console

GELU approximation accuracy:
  Maximum absolute error: 0.000473
  Mean absolute error:    0.000196

The approximation error is negligible for practical purposes. Most deep learning frameworks offer both versions, with the tanh approximation being faster on hardware that lacks specialized erf instructions.

Out[11]:

Visualization

Plot showing exact and approximate GELU curves overlapping almost perfectly, with a small inset showing the magnified error. — Comparison of exact GELU and its tanh approximation across [-4, 4]. The two curves are visually indistinguishable at this scale. The inset (magnified by 1000x) shows the approximation error, which peaks around x = 1 but remains below 0.005 across the entire range. This level of accuracy is sufficient for all practical applications.

Why GELU Became Standard for EncodersLink Copied

GELU's adoption in BERT (2018) was a turning point in natural language processing. Before BERT, most language model pretraining used simpler activations. Devlin et al. chose GELU for BERT partly based on empirical results and partly based on the theoretical appeal of smooth, probabilistic gating. The success of BERT then cemented GELU as the activation of choice for the encoder-style transformer community, including RoBERTa and ALBERT, as well as ELECTRA and virtually every BERT successor.

The connection between GELU and dropout is worth understanding. Hendrycks and Gimpel's original GELU paper framed the activation as an approximation to a network where each neuron's activation is randomly zeroed (like dropout) with probability $1 - \Phi(x)$ . From this perspective, GELU implements a kind of "stochastic depth" within each neuron's activation: high-activation neurons are nearly always kept; low-activation neurons are frequently zeroed out. This connection to regularization may partly explain why GELU trains more stably than ReLU at large scale.

In practice, the difference between GELU and a well-tuned ReLU model often comes down to training dynamics in the first few thousand steps. GELU's smooth gradients allow larger learning rates and more aggressive optimization without the risk of mass neuron death that can afflict ReLU networks. Once training is stable, both activations often achieve similar final performance. But for large models trained on large datasets, where training stability matters, GELU's smooth behavior provides meaningful advantages.