Activation Functions › Worked Example: Gradient Flow Through a Single Neuron

To make gradient flow concrete, let's trace what happens to a gradient as it passes through a single neuron with different activation functions. Suppose the neuron has a pre-activation value of $z = -0.5$ (slightly negative, a common situation early in training when weights are small and random).

For each activation, the output value and gradient are:

Sigmoid ($z = -0.5$): $\sigma(-0.5) = 1/(1 + e^{0.5}) \approx 0.378$. Gradient: $0.378 \times (1 - 0.378) \approx 0.235$. The gradient has been attenuated to 23.5% of its original value.

Tanh ($z = -0.5$): $\tanh(-0.5) \approx -0.462$. Gradient: $1 - (-0.462)^2 \approx 0.787$. The gradient retains 78.7% of its original value, much better than sigmoid.

ReLU ($z = -0.5$): $\text{ReLU}(-0.5) = 0$. Gradient: $0$. The gradient is completely blocked. This neuron contributes nothing to training for this input.

Leaky ReLU ($z = -0.5$, $\alpha = 0.01$): $\text{LeakyReLU}(-0.5) = -0.005$. Gradient: $0.01$. The gradient is tiny (1% of original) but nonzero, allowing the neuron to eventually recover.

GELU ($z = -0.5$): $\text{GELU}(-0.5) \approx -0.154$. Gradient $\approx 0.345$. The gradient retains 34.5% of its original value, substantially more than ReLU's zero.

Swish ($z = -0.5$): $\text{Swish}(-0.5) = -0.5 \times \sigma(-0.5) \approx -0.189$. Gradient $\approx 0.324$. Similar to GELU.

This single-neuron comparison captures the key tradeoff: ReLU completely blocks gradient flow for negative inputs (dying neuron problem), while smooth activations like GELU and Swish preserve a substantial fraction of the gradient, enabling recovery from initial negative activations.