Activation Functions › GELU: The Gaussian Error Linear Unit

GELU is the activation function used in BERT, GPT, and most modern transformer architectures. Understanding why it works requires examining its mechanism, which is more subtle than it might appear.

GELU combines the identity function (like ReLU's positive side) with a smooth, stochastic gating mechanism. The exact definition is:

$$\text{GELU}(z) = z \cdot \Phi(z)$$

where:

• $z$: the pre-activation value
• $\Phi(z)$: the cumulative distribution function (CDF) of the standard normal distribution, $\Phi(z) = P(X \leq z)$ for $X \sim \mathcal{N}(0, 1)$

The intuition is that GELU scales the input $z$ by the probability that a standard normal sample would be less than $z$. When $z$ is very large and positive, $\Phi(z) \approx 1$, so GELU acts like the identity. When $z$ is very large and negative, $\Phi(z) \approx 0$, so the output is nearly zero. Near $z = 0$, the gate is approximately 0.5, giving a smooth transition.

This is equivalent to saying: the neuron stochastically decides whether to pass its input or zero it out, where the probability of passing is $\Phi(z)$. Neurons with larger positive activations are more likely to be "on," and neurons with negative activations are more likely to be "off." The difference from ReLU is that the transition is smooth rather than hard, which allows gradients to flow through even slightly negative activations. Compared to a hard binary gate (as in ReLU), the smooth probabilistic gate provides more information to the optimizer about how the input contributed to the output, which translates to better gradient signal during backpropagation.

Another way to see the GELU intuition is as a regularized identity. A standard linear function $f(z) = z$ is the simplest possible activation: it passes everything through. GELU is like a linear function multiplied by a confidence score: the more strongly positive this activation is (based on its relationship to the distribution of all activations), the more of it we pass through.

The exact GELU requires computing the standard normal CDF, which involves the error function:

$$\Phi(z) = \frac{1}{2}\left[1 + \text{erf}\!\left(\frac{z}{\sqrt{2}}\right)\right]$$

where:

• $\text{erf}(x) = \frac{2}{\sqrt{\pi}} \int_0^x e^{-t^2} dt$: the error function

This is expensive to compute exactly. A fast approximation using tanh is commonly used in practice:

$$\text{GELU}(z) \approx 0.5 \cdot z \cdot \left(1 + \tanh\!\left(\sqrt{\frac{2}{\pi}} \cdot \left(z + 0.044715 z^3\right)\right)\right)$$

where:

• $0.044715$: an empirical correction constant
• $\sqrt{2/\pi} \approx 0.7978$: the normalization factor for the approximation

This approximation is accurate to within a few percent across the range that matters for neural network training and is significantly faster than computing the true error function. PyTorch provides both variants: nn.GELU() uses the exact form by default, while nn.GELU(approximate='tanh') uses the faster approximation. The original BERT paper used the tanh approximation; later work and most current implementations use the exact version.

GELU has several properties that make it well-suited to transformers.

First, the smooth gradient everywhere (no hard zero at $z = 0$) helps with the optimization landscape in deep architectures with many layers. Transformers stack dozens of identical blocks, and any activation function that creates hard discontinuities in the gradient can cause instability as gradients multiply through all those layers.

Second, GELU's slight dip into negative territory near $z \approx -0.17$ before recovering to zero produces a non-monotone shape. Unlike ReLU, which is monotone everywhere, GELU can produce slightly negative outputs even for slightly negative inputs. This non-monotone property empirically helps the network learn more complex feature representations by allowing the activation to express subtle distinctions in the negative region.

Third, the stochastic interpretation aligns well with dropout-like regularization. GELU can be understood as a smooth approximation to a neuron that randomly drops its activation based on a Bernoulli distribution parameterized by $\Phi(z)$. This provides an implicit regularization effect similar to dropout, without the discreteness and variance that dropout introduces.

BERT, GPT-2, GPT-3, and most subsequent transformer architectures use GELU or a variant. The choice has become canonical for transformers in the same way that ReLU became canonical for CNNs.