FFN Activation Functions › Limitations and Impact

The choice of activation function in transformer FFNs has subtle but measurable effects on model behavior. While the differences between GELU and SiLU are often small in benchmarks, they compound across billions of forward passes during training.

One underappreciated limitation is numerical stability. GELU's error function and SiLU's exponential can both produce numerical issues at extreme input values. Production implementations include clipping (as we did for sigmoid) and sometimes use mixed-precision computation to balance speed and stability. The clipping range matters: a range that is too tight may introduce bias by capping legitimate high-magnitude activations, while a range that is too loose may fail to prevent overflow in edge cases.

Another consideration is hardware optimization. Modern GPUs and TPUs have specialized circuits for certain operations. The sigmoid function in SiLU benefits from this optimization on many accelerators. GELU's error function is less universally optimized, which partly explains the trend toward SiLU in recent models despite GELU's theoretical elegance. When Facebook (Meta) designed the A100-optimized LLaMA architecture, the choice of SiLU over GELU was partly driven by the sigmoid's hardware-friendly properties.

The impact of activation functions extends beyond raw performance. The smooth gradients of GELU and SiLU allow for more stable training at large batch sizes and high learning rates. This mattered when scaling transformers to billions of parameters, where training instability becomes a serious concern. Models trained with ReLU show higher variance in loss curves and more sensitivity to hyperparameter choices than models trained with smooth activations. At very large scale, this training instability has real costs: a training run that diverges and must be restarted wastes compute that can cost millions of dollars.

One limitation that receives less attention is the activation function's interaction with weight initialization. Standard initialization schemes like Glorot (Xavier) or He initialization make assumptions about the activation function's expected output variance. He initialization, for example, is derived specifically for ReLU by accounting for the fact that half of ReLU's outputs are zero. Using He initialization with GELU or SiLU produces slightly incorrect scaling, since those activations do not zero out exactly half of their inputs. This mismatch is usually small enough not to matter in practice, especially with modern normalization schemes like LayerNorm, but it illustrates that the choice of activation cannot be fully decoupled from other initialization and normalization decisions.

A deeper limitation is our incomplete theoretical understanding of why smooth activations outperform ReLU at large scale. The empirical evidence is consistent and reproducible: GELU and SiLU generally work better than ReLU for large language models. But the precise mechanism remains incompletely understood. Is it the smoothness? The non-monotonicity? The probabilistic interpretation? The negative dip? Ablation studies can test individual hypotheses, but the interactions between activation choice and initialization, plus normalization and optimizer, are complex enough that clean causal explanations are hard to establish. This uncertainty is a reminder that even core architectural choices in modern deep learning rest partly on empirical foundations rather than pure theory.

Looking forward, activation functions continue to evolve. Gated linear units (covered in the next chapter) combine activation functions with multiplicative gating, creating even richer nonlinearities. The field hasn't settled on a final answer, and future architectures may introduce entirely new activation functions that further improve on the current options. There is ongoing research into activation functions with adaptive behavior, where the function's shape is conditioned on context or learned per-layer. Whether such approaches will yield practical improvements beyond SiLU and GELU at current scales remains an open question.