Activation Functions › Limitations and Practical Considerations

No activation function is perfect for all situations. Understanding the failure modes helps you debug training problems and make better architecture choices.

The dying ReLU problem remains the most common activation-related training failure. Networks trained with large learning rates, without proper weight initialization, or with many layers are most susceptible. If you observe that training loss stops improving early in training and learning rate adjustments do not help, checking the fraction of dead neurons is a useful diagnostic step. A simple approach is to run the entire training set through the model and check how many neurons output zero for all inputs, as shown in the code above. If more than 20-30% of neurons are dead, switching to Leaky ReLU or GELU is likely to help.

GELU and Swish are computationally more expensive than ReLU due to the sigmoid or error function computation. For latency-critical inference applications, this overhead can matter. On modern GPU hardware, the difference is usually negligible because the computation is memory-bandwidth-bound rather than compute-bound. However, on CPU inference or edge devices, the transcendental function evaluations (exponential, tanh) can be a real bottleneck. Profiling on your target hardware before committing to a smooth activation is good practice.

SELU's self-normalization property only holds under strict conditions. Batch normalization, dropout, or skip connections all break the statistical invariant that SELU relies on. Using SELU in a standard ResNet without removing these components is counterproductive and will not provide the self-normalization benefit. SELU is best suited to pure feedforward architectures where you have full control over the normalization strategy.

For activation functions in quantized or pruned models (where weights are represented in low precision), smooth activations like GELU can cause accuracy degradation because the slight negative values are lost when the network is quantized to integers. ReLU's hard zero is quantization-friendly: the threshold at zero is a natural fit for integer quantization schemes. If you need to quantize a model after training, testing whether switching from GELU to ReLU before quantization improves post-quantization accuracy is worth doing.

The relationship between activation functions and batch normalization is also important. Batch normalization is often applied before the activation function (in post-norm designs like the original ResNet) or after it (in some transformer variants). The position of batch normalization relative to the activation function changes the effective input distribution that the activation sees. If batch normalization is applied before the activation, the activation always sees inputs that are approximately standardized (mean 0, variance 1), which means it always operates near $z = 0$ where its gradient is at its highest. This reduces the saturation problem for sigmoid and tanh, making them more viable with batch normalization than without. Pre-norm transformers, which apply layer normalization before the attention and feed-forward sub-layers, benefit from a similar effect.

Finally, the activation function interacts with the weight initialization strategy. He initialization ($\text{variance} = 2/n$) is designed for ReLU, accounting for the fact that ReLU zeros out roughly half of its inputs. For GELU or Swish, a slightly different variance is theoretically optimal, but in practice He initialization works well enough for all the smooth activations. Glorot (Xavier) initialization is better suited to sigmoid and tanh and tends to underperform for ReLU-like functions.