Activation Functions › Activation Function Comparison

To compare these functions, it helps to look at four properties: the output range, whether the function is zero-centered, the gradient at zero, and the computational cost.

To understand how these functions affect gradient flow in deep networks, consider what happens when you backpropagate a gradient $\delta$ through a layer with activation $f$. The gradient signal that reaches the layer below is $\delta \cdot f'(z)$. If $f'(z) < 1$ consistently, the gradient shrinks at each layer. If $f'(z) > 1$, it grows (which can cause exploding gradients).

For sigmoid, $f'(z) \leq 0.25$ everywhere. Backpropagating through $N$ sigmoid layers multiplies the gradient by at most $0.25^N$. With just 10 layers, this is $(0.25)^{10} \approx 10^{-6}$: a million-fold reduction. The early layers receive essentially no gradient and their weights barely move.

For ReLU, $f'(z) = 1$ on the active side. Backpropagating through $N$ active ReLU neurons multiplies the gradient by $1^N = 1$: no shrinkage at all. This is the core reason ReLU enabled training of deep networks. The caveat is that dead neurons (the $f'(z) = 0$ side) block gradient flow entirely, which is why the dying ReLU problem matters for deep or poorly initialized networks.

For GELU and Swish, the derivative near zero is approximately 0.5, and the gradient grows to approximately 1 for large positive inputs. These smooth functions have slightly smaller average gradients than ReLU on the positive side but handle the transition at zero more gracefully, avoiding the hard zero that causes dying neurons. The gradient for GELU varies between approximately 0.5 and 1.0 for typical pre-activation values, which means GELU networks do experience some gradient attenuation but far less than sigmoid networks.

The following plot illustrates gradient magnitude decay across layers for sigmoid versus ReLU, showing the dramatic difference that activation choice makes in deep networks:

In controlled experiments on standard benchmarks, the relative convergence speed of activation functions is roughly:

GELU $\approx$ Swish $\approx$ Mish $>$ ELU $>$ Leaky ReLU $\approx$ ReLU $>$ tanh $>$ sigmoid

This ordering is approximate and architecture-dependent. The modern smooth functions (GELU, Swish, Mish) often converge faster and to slightly better optima, particularly in deeper networks. The advantage over ReLU is often small (a few percent in final accuracy) but consistent enough across benchmarks that transformers standardized on GELU.

The reason the smooth activations tend to converge faster is subtle. ReLU creates a piecewise-linear loss landscape: the loss surface is flat in all the directions where neurons are dead. This means gradient descent must take many small steps around the flat regions. GELU and Swish produce a smoother landscape with fewer flat regions, so gradient descent can move more directly toward the optimum. The difference is not dramatic for shallow networks, but compounds over many layers.