FFN Activation Functions: ReLU, GELU

The feed-forward network's power comes from its nonlinear activation function. Without nonlinearity, stacking multiple linear layers would collapse into a single linear transformation, no matter how deep the network. The activation function is what enables FFNs to approximate arbitrarily complex functions of language.

Think of an activation function as the network's ability to "make decisions." A purely linear transformation can only scale and rotate data; it can express linear relationships but nothing more. Introduce even a single nonlinear function, and the network can learn curves and thresholds alongside interactions. This distinction, between linear and nonlinear computation, separates a powerful learning machine from a sophisticated calculator.

The key insight is that activation functions do not just inject nonlinearity. They also control how gradients flow backward through the network during training. A poorly chosen activation can starve layers of gradient signal, causing the network to stop learning. The history of deep learning is partly a history of discovering better activation functions, each addressing a specific failure mode of its predecessor.

The original transformer used ReLU, the workhorse of deep learning since 2012. But as researchers scaled transformers to billions of parameters and trained them on trillions of tokens, subtle differences between activation functions became significant. GELU emerged as the standard for encoder models like BERT. SiLU (also called Swish) now dominates decoder models like LLaMA and GPT-NeoX. Understanding why these activations differ, and when each excels, is essential for building modern language models.

In practice, you will rarely train a language model from scratch and choose its activation function freely. You will more often be fine-tuning a model that already has a baked-in activation choice. But understanding the reasoning behind these choices helps you interpret model behavior, diagnose training problems, and make informed decisions when you do have architectural freedom. It also helps you read the research literature, where activation functions appear as key variables in ablation studies.

This chapter traces the evolution from ReLU to GELU to SiLU/Swish, examining the mathematical properties that make each function suitable for different contexts. You'll implement each activation, visualize their differences, and understand the practical trade-offs that guide model design choices.

Before examining specific activation functions, it is worth building a deeper intuition for why nonlinearity matters so much. The feed-forward sublayer in a transformer takes a vector of dimension $d_{\text{model}}$, projects it up to a higher-dimensional space of size $d_{\text{ff}}$, applies an activation, and then projects back down. In mathematical terms, a two-layer FFN computes:

where:

• $\mathbf{x} \in \mathbb{R}^{d_{\text{model}}}$: the input vector from the attention sublayer
• $W_1 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$: the first projection (expands dimensionality)
• $b_1 \in \mathbb{R}^{d_{\text{ff}}}$: the first bias vector
• $\text{act}(\cdot)$: the activation function applied elementwise
• $W_2 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$: the second projection (reduces dimensionality back)
• $b_2 \in \mathbb{R}^{d_{\text{model}}}$: the second bias vector

If $\text{act}$ were the identity function (no nonlinearity), the composition $W_2 \cdot (W_1 \mathbf{x} + b_1) + b_2$ would simplify to $W_2 W_1 \mathbf{x} + W_2 b_1 + b_2$, which is just a linear transformation. Stacking twenty such layers would still produce one linear transformation. All that computation would buy you nothing beyond a single matrix multiply.

The universal approximation theorem tells us that a network with a single hidden layer and a nonlinear activation can approximate any continuous function to arbitrary precision, given enough hidden units. The key requirement is exactly that nonlinearity. In practice, deeper networks with smaller hidden layers learn more efficiently than shallow networks with enormous hidden layers, but the theoretical foundation is the same: nonlinearity is the ingredient that maps matrix multiplication into universal function approximation.

For language modeling specifically, the FFN layers are thought to store factual knowledge. Research by Geva et al. (2021) showed that individual FFN layers function like key-value memories: the first layer's columns act as "keys" that match patterns in the input, and the second layer's rows act as "values" that contribute to the output distribution. The activation function is what allows these keys to be selective, activating strongly for certain input patterns and weakly for others. Without nonlinearity, every key would always partially match every input, making the retrieval process incoherent.

Notice that the FFN's nonlinearity is fundamentally different from attention's nonlinearity. Attention creates nonlinear interactions between positions (which tokens attend to which). The FFN creates nonlinear transformations within each position independently. Together, these two nonlinearities give the transformer its expressive power.

The Rectified Linear Unit (ReLU) is the simplest nonlinear activation function in modern deep learning. It applies a trivial rule: keep positive values unchanged, set negative values to zero. Mathematically:

where:

• $x$: the input value (a single element of the pre-activation vector)
• $\max(0, x)$: returns $x$ if $x > 0$, otherwise returns $0$

This simplicity is deceptive. ReLU revolutionized deep learning when introduced in AlexNet (2012), enabling training of much deeper networks than the sigmoid and tanh activations that preceded it. The original transformer adopted ReLU for its feed-forward layers, inheriting this proven workhorse.

Think of ReLU as a one-way valve. Values trying to flow in the positive direction pass through without resistance. Values trying to flow in the negative direction are blocked completely. The network learns to use this valve to select which "features" in the hidden representation are worth propagating and which should be silenced.

The key insight behind ReLU's success is its effect on gradients. Sigmoid and tanh both saturate: for very large or very small inputs, their derivatives approach zero, making it nearly impossible for deep layers to receive useful gradient signal. ReLU does not saturate for positive inputs. Its derivative is exactly 1 for any positive value, meaning gradient flows through positive activations without shrinkage. This simple property allowed researchers to train networks twenty, fifty, even one hundred layers deep, something that had been essentially impossible with saturating activations.

Let's implement ReLU and examine its behavior:

ReLU's advantages are clear. Its computation is trivial: a single comparison operation. The derivative is equally simple:

where:

• The derivative is 1 for positive inputs, meaning gradients flow through unchanged
• The derivative is 0 for negative inputs, meaning gradients are blocked entirely
• The derivative is technically undefined at exactly $x = 0$, but implementations typically use 0 or 1

This step function gradient never vanishes for positive inputs like sigmoid's gradient does for large values. And ReLU creates sparse activations, as roughly half of all hidden units output zero for typical input distributions, which can improve efficiency and interpretability.

In practice, the sparsity ReLU creates is both a feature and a design principle. Sparse representations are more interpretable: when only a subset of neurons activate for a given input, you can potentially trace which features are "firing" for which inputs. Sparse activations also enable optimizations in hardware, since multiplying by zero can be skipped. In very large FFN layers (some models use $d_{\text{ff}} = 16{,}384$ or larger), exploiting this sparsity can reduce compute.

But ReLU has a critical weakness. When a neuron's pre-activation is consistently negative, its gradient is always zero. The neuron stops learning entirely. This "dying ReLU" problem becomes more severe as networks deepen or learning rates increase.

Consider what happens during training. If a weight update pushes a neuron's bias too negative, that neuron's output becomes zero for all inputs. With zero output, the gradient flowing through that neuron is zero. With zero gradient, the weights never update. The neuron is dead.

The cascading nature of this problem makes it particularly insidious. A dead neuron in an early layer of the FFN prevents any downstream computation from receiving information through that pathway. If many neurons die, the effective width of the network shrinks, reducing its capacity. In extreme cases, if a whole layer becomes largely dead, the model trains without using that layer at all, wasting parameters and compute.

Dead neurons arise most commonly from two causes. The first is a high initial learning rate: a large gradient step early in training can push many neurons into the negative half-plane simultaneously, and if the learning rate remains high, the neurons never recover. The second is weight initialization that creates very negative initial biases, giving neurons little chance of activating before training has even begun. Careful initialization (like He initialization, which accounts for ReLU's expected output variance) reduces the risk but does not eliminate it.

The plot shows how quickly neurons die as bias values shift negative. In real training, this can happen in patches of the network, reducing effective capacity without obvious symptoms. The model trains, but parts of it have effectively disconnected from the computation.

Notice that the dying-neuron problem has no easy diagnostic signal during training. The loss might still decrease, the validation metrics might still improve, but you could be doing so with only a fraction of the network's intended capacity. This silent degradation motivated researchers to search for activations that provide nonzero gradients everywhere, even for negative inputs.

Several ReLU variants attempted to address the dead neuron problem before GELU and SiLU became the dominant alternatives. Leaky ReLU uses a small negative slope (typically 0.01) for negative inputs instead of zeroing them:

where:

• $\alpha$: a small positive constant (typically 0.01), sometimes called the "leak" coefficient

By allowing a tiny gradient for negative inputs, Leaky ReLU prevents neurons from dying completely. Parametric ReLU (PReLU) takes this further by making $\alpha$ a learnable parameter, allowing the network to decide how much of the negative signal to preserve. ELU (Exponential Linear Unit) uses a smooth exponential function for negative inputs, avoiding the discontinuity entirely.

These variants were steps in the right direction, but none of them fully addressed the deeper issue: ReLU's sharp corner at zero creates an abrupt transition that can destabilize training in large models. Researchers ultimately needed a non-zero negative gradient inside a smooth, differentiable function throughout its entire domain. That insight led to GELU.

Hendrycks and Gimpel introduced the Gaussian Error Linear Unit (GELU) in 2016, though it didn't gain widespread adoption until BERT popularized it in 2018. GELU addresses ReLU's sharp corner by introducing smooth, probabilistic gating.

The intuition behind GELU is elegant: instead of deterministically zeroing negative values, gate each input by its probability of being positive under a standard normal distribution. Inputs that are clearly positive (large positive values) pass through nearly unchanged. Inputs that are clearly negative (large negative values) are nearly zeroed. Inputs near zero, where classification is uncertain, are partially attenuated.

Think of GELU as a soft decision boundary instead of a hard one. Whereas ReLU says "if you're negative, you're blocked, full stop," GELU says "if you're probably negative, you're mostly blocked, but we'll keep a little signal proportional to how uncertain we are." This probabilistic framing connects activation functions to Bayesian reasoning: the gate is literally a probability estimate.

The mathematical formulation follows this intuition:

where:

• $x$: the input value to the activation function
• $\Phi(x)$: the cumulative distribution function (CDF) of the standard normal distribution, representing the probability that a standard normal random variable is less than or equal to $x$

The CDF itself is computed using the error function:

where:

• $P(X \leq x)$: the probability that a standard normal random variable $X$ takes a value less than or equal to $x$
• $\text{erf}(\cdot)$: the Gauss error function, a special function that arises in probability and statistics
• $\sqrt{2}$: a scaling factor that converts from the standard normal to the error function's parameterization

The CDF $\Phi(x)$ ranges from 0 to 1. For large positive $x$, $\Phi(x) \approx 1$, so $\text{GELU}(x) \approx x$. For large negative $x$, $\Phi(x) \approx 0$, so $\text{GELU}(x) \approx 0$. The transition is smooth, governed by the bell curve of the normal distribution.

The key insight here is the choice of the normal distribution as the gating mechanism. By using $\Phi(x)$, we are essentially asking: "if I draw a random variable from a standard normal, how likely is it to be less than this input?" For a pre-activation value of 0, the answer is 50%, so GELU attenuates zero-valued inputs by half. For a pre-activation value of 2, the answer is about 97.7%, so GELU passes almost all of that signal. This creates a natural, data-independent scale for "how much" to pass each value.

To build concrete intuition, let's trace a single pre-activation value through the GELU computation step by step. Suppose the FFN's first layer produces a pre-activation of $x = 1.0$ for a particular neuron.

Step 1: Compute the CDF value at $x = 1.0$. The standard normal CDF at 1.0 is $\Phi(1.0) \approx 0.8413$. This says: "there is an 84.1% probability that a standard normal variable is at most 1.0."

Step 2: Multiply the input by this CDF value:

The neuron passes about 84% of its signal. Now suppose the pre-activation is $x = -1.0$. The CDF value is $\Phi(-1.0) \approx 0.1587$, so:

The neuron passes about 16% of its (negative) signal, attenuating it substantially but not zeroing it. Compare this to ReLU, which would output 0 for $x = -1.0$ and 1.0 for $x = 1.0$. GELU's output is softer in both cases: positive activations are attenuated slightly, negative activations are attenuated heavily but remain nonzero.

This soft gating behavior, controlled by the Gaussian CDF, is what makes GELU particularly well-suited to the pre-activation distributions that arise in transformer FFNs. Transformer pre-activations after LayerNorm are approximately normally distributed. GELU's gating function is calibrated to the same normal distribution, making the soft-gating particularly well-aligned with the actual statistical properties of the data it operates on.

Computing the exact error function is expensive. In practice, transformers use fast approximations. The most common is the tanh approximation:

where:

• $x$: the input value
• $\tanh(\cdot)$: the hyperbolic tangent function, which outputs values in the range $(-1, 1)$
• $\sqrt{2/\pi} \approx 0.7979$: a scaling constant derived from properties of the normal distribution
• $0.044715$: an empirically-fitted coefficient that improves the approximation accuracy
• $x + 0.044715 x^3$: a polynomial that approximates the argument to the error function

This formula looks complex, but it avoids the expensive erf computation while matching GELU closely. The constants were determined by fitting to the exact GELU curve, minimizing the maximum error across typical input ranges.

The reason this approximation works so well is that $\tanh$ and $\text{erf}$ are closely related functions. Both are odd functions (symmetric about the origin), both map the real line to $(-1, 1)$, and both have an S-shaped curve. The polynomial $x + 0.044715 x^3$ inside the $\tanh$ call adjusts the rate of transition to closely match the CDF shape. The result is an approximation that differs from the exact GELU by at most about 0.004 across the full input range.

Let's compare the exact and approximate versions:

GELU's adoption in BERT (2018) was a turning point in natural language processing. Before BERT, most language model pretraining used simpler activations. Devlin et al. chose GELU for BERT partly based on empirical results and partly based on the theoretical appeal of smooth, probabilistic gating. The success of BERT then cemented GELU as the activation of choice for the encoder-style transformer community, including RoBERTa and ALBERT, as well as ELECTRA and virtually every BERT successor.

The connection between GELU and dropout is worth understanding. Hendrycks and Gimpel's original GELU paper framed the activation as an approximation to a network where each neuron's activation is randomly zeroed (like dropout) with probability $1 - \Phi(x)$. From this perspective, GELU implements a kind of "stochastic depth" within each neuron's activation: high-activation neurons are nearly always kept; low-activation neurons are frequently zeroed out. This connection to regularization may partly explain why GELU trains more stably than ReLU at large scale.

In practice, the difference between GELU and a well-tuned ReLU model often comes down to training dynamics in the first few thousand steps. GELU's smooth gradients allow larger learning rates and more aggressive optimization without the risk of mass neuron death that can afflict ReLU networks. Once training is stable, both activations often achieve similar final performance. But for large models trained on large datasets, where training stability matters, GELU's smooth behavior provides meaningful advantages.

While GELU became the activation of choice for encoder models like BERT, a different function emerged for large decoder models: SiLU, also known as Swish. Google researchers introduced Swish in 2017 through neural architecture search, discovering that this simple formula consistently outperformed ReLU across tasks.

where:

• $x$: the input value
• $\sigma(x)$: the logistic sigmoid function (defined below)
• $e^{-x}$: the exponential function with negative argument, which rapidly approaches 0 for large positive $x$ and infinity for large negative $x$

The sigmoid function that gates the input is:

where:

• $\sigma(x)$: outputs a value in the range $(0, 1)$, interpretable as a "gate" or probability
• For large positive $x$: $\sigma(x) \to 1$, so $\text{SiLU}(x) \to x$ (input passes through)
• For large negative $x$: $\sigma(x) \to 0$, so $\text{SiLU}(x) \to 0$ (input is suppressed)
• At $x = 0$: $\sigma(0) = 0.5$, so $\text{SiLU}(0) = 0$

The name "SiLU" stands for Sigmoid Linear Unit. This stresses its structure: the input multiplied by its sigmoid. "Swish" was Google's original branding. The two names refer to the same function, though the literature isn't always consistent.

Notice the structural similarity between SiLU and GELU. Both functions multiply the input $x$ by a "gate" value that lies between 0 and 1. In GELU, the gate is $\Phi(x)$, the Gaussian CDF. In SiLU, the gate is $\sigma(x)$, the logistic sigmoid. The Gaussian CDF and the logistic sigmoid are very similar functions: both are S-shaped, both map the real line to $(0, 1)$, and both equal 0.5 at $x = 0$. The key difference is their transition rate: the logistic sigmoid transitions slightly more quickly, leading to SiLU's slightly deeper negative dip compared to GELU.

Let's trace the same pre-activation values through SiLU to compare with GELU. For $x = 1.0$:

For $x = -1.0$:

Compare these to GELU's values: GELU(1.0) $\approx$ 0.841 and GELU(-1.0) $\approx$ -0.159. SiLU attenuates positive inputs more (0.731 vs. 0.841) and amplifies negative inputs more (-0.269 vs. -0.159). The logistic sigmoid transitions more quickly than the Gaussian CDF, which explains SiLU's larger negative dip and slightly sharper transition region.

In practice, these differences are small enough that switching between GELU and SiLU rarely changes model quality measurably unless you are training at very large scale. The choice between them is largely a matter of convention and ecosystem: GELU for BERT-like models, SiLU for LLaMA-like models.

The preference for SiLU in decoder models like LLaMA and Mistral, as well as Falcon, isn't fully understood theoretically, but several factors contribute:

1. Simpler computation: SiLU requires only sigmoid and multiplication, avoiding the error function or its approximations. On modern hardware with fast sigmoid implementations, this can be faster than GELU.

2. Slightly stronger gradients: SiLU's derivative can exceed 1 (peaking around 1.1 near $x = 1.5$), which may help gradient flow in very deep networks. GELU's derivative is bounded between 0 and 1.

3. Empirical performance: In large-scale experiments at the scale of modern LLMs, SiLU consistently matches or slightly outperforms GELU for autoregressive language modeling. The differences are small but measurable.

4. Pairing with gated linear units: Modern architectures like LLaMA use SiLU specifically within gated linear unit (GLU) variants, where the activation's properties interact with the gating mechanism. We'll explore this in the next chapter.

The historical sequence helps explain this preference. When Meta released LLaMA in early 2023 and described using SiLU in their SwiGLU-based FFN, SiLU's adoption accelerated rapidly across the open-source community. Mistral and Falcon, along with Gemma and Phi, were among the subsequent open models that followed the LLaMA architecture closely, including its SiLU activation. This created a strong network effect: the ecosystem of model weights, fine-tuning code, and inference libraries all assumed SiLU, making it the path of least resistance for new decoder models.

With all three activations implemented, let's visualize them together to understand their relationships and differences:

Let's compute specific properties that affect neural network behavior:

Key observations from this analysis:

• Minimum value: ReLU never goes negative (by definition). GELU's dip is mild (-0.17), while SiLU dips deeper (-0.28). These negative values can help with regularization but also introduce potential instabilities.

• Sparsity: All three create some degree of sparsity (outputs near zero), but ReLU is the most sparse for normally distributed inputs. This sparsity can aid interpretability and efficiency.

• Linearity: All three are highly linear for positive inputs (correlation > 0.999 with $y = x$), which helps preserve signal magnitude through deep networks.

The sparsity difference between ReLU and the smooth activations is worth dwelling on. For a standard normal input distribution, ReLU produces exactly 50% sparsity by zeroing all negative values. GELU and SiLU produce much less sparsity because they only attenuate negative inputs, not zero them. This means GELU and SiLU FFN layers operate with denser intermediate representations, which requires slightly more memory bandwidth during training and inference.

GELU and SiLU's negative regions deserve closer examination. For certain negative inputs, these activations produce negative outputs rather than zero. This seems counterintuitive: why would we want activations to sometimes invert their input's sign?

The negative dip is a form of implicit regularization. Consider what happens during training: inputs that land in the dip region (roughly $-2 < x < 0$ for GELU) get attenuated but not eliminated. This "soft gating" provides a richer gradient signal than ReLU's hard zero, potentially helping the network escape local minima.

Think of it this way: a neuron in a deep network exists among neighboring neurons, all contributing to the next layer's computation. If one neuron's activation is slightly negative, zeroing it (as ReLU does) eliminates any information that neuron might carry. Allowing it to pass a small negative value (as GELU and SiLU do) preserves that signal, even if attenuated. The downstream layer can then decide whether to amplify or further attenuate it.

The key insight is that the sign of a pre-activation carries information. A pre-activation of $-0.3$ is different from a pre-activation of $-3.0$ in a meaningful way: the first suggests the input is borderline, while the second suggests a strong mismatch with the learned pattern. GELU and SiLU preserve this distinction by scaling the output proportionally. ReLU treats both identically: output zero.

The magnitude of the dip matters for training dynamics. A deeper dip (SiLU) means stronger negative signals for moderately negative inputs, which could either help or hurt depending on the task. Empirically, both work well, with the choice often coming down to which pairs better with other architectural decisions (like gated linear units).

There is also a connection between the negative dip and the activation's non-monotonicity. A monotonic function never decreases as its input increases: ReLU and the standard sigmoid are monotonic. GELU and SiLU are non-monotonic: they decrease slightly in the range around $x \approx -0.5$ before increasing again. This non-monotonicity is unusual for activation functions and was initially viewed with some skepticism. The practical evidence, however, is that non-monotonic activations train comparably to or better than monotonic alternatives at large scale, suggesting that the strict monotonicity constraint imposed by ReLU and its variants was unnecessarily restrictive.

Let's simulate how these activations behave in an actual FFN layer, processing typical transformer hidden states:

Activation function speed matters at scale. When processing billions of tokens through trillion-parameter models, even small differences in activation computation time accumulate. The helper below shows how to benchmark the implementations. For the visualization, we use one fixed reference measurement so the data remains identical across light, dark, and transparent render variants:

In this reference measurement, ReLU is fastest because it is just a comparison operation. The exact GELU using erf is slowest. The GELU tanh approximation is faster than exact GELU but still slower than SiLU. SiLU is relatively fast because sigmoid has efficient implementations on most hardware.

In practice, these differences are often dwarfed by memory bandwidth limitations and matrix multiplication costs. The activation function typically accounts for less than 1% of total FFN compute time. Still, at sufficient scale, even small improvements matter.

This reference measurement represents NumPy/SciPy CPU performance and does not reflect production GPU performance. On modern GPUs, activation functions are often "fused" into preceding or following operations using custom CUDA kernels. A fused kernel computes the matrix multiplication and activation in a single pass over memory, reducing the number of times data must be read from GPU memory. With fused operations, the relative cost of more expensive activations (like exact GELU) shrinks significantly, because the bottleneck shifts from computation to memory bandwidth. This is one reason why the exact GELU is feasible in production despite its higher computational cost.

The choice of activation function depends on your model architecture and use case. Here's a practical guide:

Use ReLU when:

• You need maximum computational efficiency and simplicity
• You're building a custom architecture and want a well-understood baseline
• Your model is small enough that the dead neuron problem is manageable
• You're working with older codebases or frameworks that expect ReLU

Use GELU when:

• You're building an encoder model (BERT, RoBERTa, ELECTRA style)
• You want compatibility with pretrained encoder models
• You prioritize smooth gradients over computational efficiency
• You're fine-tuning an existing GELU-based model

Use SiLU when:

• You're building a decoder model (GPT, LLaMA, Mistral style)
• You're using gated linear units (SwiGLU, GeGLU) in your FFN
• You want the slight performance edge that modern LLMs have demonstrated
• You need a smooth activation but prefer simpler computation than GELU

The empirical differences between GELU and SiLU are often small. Unless you're training at massive scale where every fraction of a percent matters, either smooth activation will likely work well. The more important choice is between ReLU (with its dead neuron risk) and the smooth alternatives.

When fine-tuning a pretrained model, you should essentially always match the activation used during pretraining. Changing from GELU to SiLU in the middle of a model's lifecycle would invalidate the learned weight distributions, since those weights were optimized under the original activation's gradient behavior. The weights are not activation-function agnostic: they encode information in a way that is calibrated to how the original activation transforms inputs. Switching activations after pretraining would require substantial additional training to re-adapt all the weights.

Let's create a configurable activation function module that supports all three options, following patterns used in modern transformer libraries:

In practice, PyTorch makes this even simpler. You can specify the activation using torch.nn.functional.gelu, torch.nn.functional.silu, or torch.nn.ReLU(). Hugging Face's Transformers library defines an ACT2FN dictionary that maps string names like "gelu", "silu", and "relu" to their corresponding functions, allowing model architectures to be configured via a single string parameter in the model config. This pattern lets you swap activations without changing the model code, which is convenient for ablation experiments.

The choice of activation function in transformer FFNs has subtle but measurable effects on model behavior. While the differences between GELU and SiLU are often small in benchmarks, they compound across billions of forward passes during training.

One underappreciated limitation is numerical stability. GELU's error function and SiLU's exponential can both produce numerical issues at extreme input values. Production implementations include clipping (as we did for sigmoid) and sometimes use mixed-precision computation to balance speed and stability. The clipping range matters: a range that is too tight may introduce bias by capping legitimate high-magnitude activations, while a range that is too loose may fail to prevent overflow in edge cases.

Another consideration is hardware optimization. Modern GPUs and TPUs have specialized circuits for certain operations. The sigmoid function in SiLU benefits from this optimization on many accelerators. GELU's error function is less universally optimized, which partly explains the trend toward SiLU in recent models despite GELU's theoretical elegance. When Facebook (Meta) designed the A100-optimized LLaMA architecture, the choice of SiLU over GELU was partly driven by the sigmoid's hardware-friendly properties.

The impact of activation functions extends beyond raw performance. The smooth gradients of GELU and SiLU allow for more stable training at large batch sizes and high learning rates. This mattered when scaling transformers to billions of parameters, where training instability becomes a serious concern. Models trained with ReLU show higher variance in loss curves and more sensitivity to hyperparameter choices than models trained with smooth activations. At very large scale, this training instability has real costs: a training run that diverges and must be restarted wastes compute that can cost millions of dollars.

One limitation that receives less attention is the activation function's interaction with weight initialization. Standard initialization schemes like Glorot (Xavier) or He initialization make assumptions about the activation function's expected output variance. He initialization, for example, is derived specifically for ReLU by accounting for the fact that half of ReLU's outputs are zero. Using He initialization with GELU or SiLU produces slightly incorrect scaling, since those activations do not zero out exactly half of their inputs. This mismatch is usually small enough not to matter in practice, especially with modern normalization schemes like LayerNorm, but it illustrates that the choice of activation cannot be fully decoupled from other initialization and normalization decisions.

A deeper limitation is our incomplete theoretical understanding of why smooth activations outperform ReLU at large scale. The empirical evidence is consistent and reproducible: GELU and SiLU generally work better than ReLU for large language models. But the precise mechanism remains incompletely understood. Is it the smoothness? The non-monotonicity? The probabilistic interpretation? The negative dip? Ablation studies can test individual hypotheses, but the interactions between activation choice and initialization, plus normalization and optimizer, are complex enough that clean causal explanations are hard to establish. This uncertainty is a reminder that even core architectural choices in modern deep learning rest partly on empirical foundations rather than pure theory.

Looking forward, activation functions continue to evolve. Gated linear units (covered in the next chapter) combine activation functions with multiplicative gating, creating even richer nonlinearities. The field hasn't settled on a final answer, and future architectures may introduce entirely new activation functions that further improve on the current options. There is ongoing research into activation functions with adaptive behavior, where the function's shape is conditioned on context or learned per-layer. Whether such approaches will yield practical improvements beyond SiLU and GELU at current scales remains an open question.

Activation functions inject nonlinearity into the feed-forward network, enabling transformers to learn complex functions of language. This chapter traced the evolution from ReLU to GELU to SiLU, examining the mathematical properties that make each suitable for different contexts.

Key takeaways:

• ReLU ($\max(0, x)$) is the simplest activation, with zero computation cost beyond a comparison. Its sharp corner creates sparse representations but risks "dead neurons" that stop learning entirely. ReLU was used in the original transformer but has been largely superseded in modern architectures.

• GELU ($x \cdot \Phi(x)$) smoothly gates inputs by their Gaussian CDF values. The result is a differentiable, non-monotonic function with a slight negative dip around $x \approx -0.5$. GELU became the standard for encoder models (BERT, RoBERTa) and remains widely used.

• SiLU/Swish ($x \cdot \sigma(x)$) multiplies inputs by their sigmoid values. Like GELU, it's smooth and non-monotonic, but with a deeper negative dip and slightly simpler computation. SiLU has become the standard for decoder models (LLaMA, Mistral, GPT-NeoX).

• Practical choice: The differences between GELU and SiLU are often small in practice. Choose based on your model family (encoder vs. decoder), compatibility with pretrained models, or specific architectural requirements like gated linear units.

• Computational efficiency: ReLU is fastest, followed by SiLU, then GELU approximation, then exact GELU. However, activation computation is typically less than 1% of total FFN time, so speed rarely drives the choice.

• Never change activations during fine-tuning: Pretrained model weights encode information calibrated to the original activation. Switching activations mid-lifecycle requires retraining to re-adapt the weights.

The next chapter examines gated linear units (GLUs), which combine activation functions with multiplicative gating to create even more expressive FFN architectures. Variants like SwiGLU and GeGLU have become standard in state-of-the-art models like LLaMA and PaLM.

When configuring activation functions for transformer FFNs, these parameters and choices affect model behavior:

• activation_type: The activation function to use. Common options are "relu", "gelu", "gelu_approximate", and "silu". Choose based on your model architecture: GELU for encoder models (BERT-style), SiLU for decoder models (LLaMA-style), or ReLU for maximum simplicity and speed.

• approximate (for GELU): Whether to use the tanh approximation instead of the exact error function. The approximation is faster on most hardware and introduces negligible error (< 0.005 maximum). Most production systems use the approximation.

• inplace (framework-specific): Some frameworks allow in-place activation to reduce memory usage. This modifies the input tensor directly rather than creating a new output tensor. Use with caution, as it can cause issues with gradient computation if the original values are needed.

• numerical clipping (for SiLU/sigmoid): Input values should be clipped to prevent overflow in the exponential function. A range of $[-500, 500]$ is typical. Most deep learning frameworks handle this automatically, but custom implementations should include explicit clipping.

• dtype considerations: Activation functions behave differently at different precisions. At float16 or bfloat16, the dynamic range is limited, which can cause issues with the exponential in SiLU or the error function in GELU. Mixed-precision training typically keeps activations in higher precision for stability.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about activation functions in transformer feed-forward networks.