Backpropagation › Gradient Flow Through Activation Functions

The backward pass depends critically on the derivatives of activation functions. Different activations have very different gradient behaviors, which directly affects how well information flows backward through deep networks. We covered activation functions in depth in the Activation Functions chapter, so here we focus on their gradient properties specifically in the context of backpropagation.

The sigmoid function $\sigma(z) = \frac{1}{1+e^{-z}}$ has derivative:

$$\sigma'(z) = \sigma(z)(1 - \sigma(z))$$

where:

• $\sigma(z)$: the sigmoid output, a value in $(0, 1)$
• $\sigma'(z)$: the derivative, which equals $\sigma(z)(1-\sigma(z))$

The maximum value of $\sigma'(z)$ is $0.25$, achieved at $z = 0$. For $|z|$ large, the sigmoid saturates (output near 0 or 1), and the derivative approaches 0. This vanishing gradient problem is severe: each sigmoid layer multiplies the gradient by at most 0.25. In a 10-layer network, the gradient at the first layer could be multiplied by $0.25^{10} \approx 10^{-6}$ relative to the last layer.

There is another way to see why this is so problematic. The gradient of the loss with respect to the first layer's weights is computed as a product of gradients across all intermediate layers. If those gradients are small fractions (as they are for sigmoid in saturation), their product is tiny. This means the first-layer weights barely move during training. The network effectively cannot learn any useful feature representation in its early layers when trained with sigmoid activations in deep networks.

The hyperbolic tangent $\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$ has derivative:

$$\tanh'(z) = 1 - \tanh^2(z)$$

This has maximum value 1.0 at $z = 0$ and decays toward 0 as $|z|$ grows. Tanh is better than sigmoid in that its maximum derivative is 1 (versus 0.25 for sigmoid), but it still saturates and can cause vanishing gradients in deep networks. The fact that tanh is zero-centered (outputs range from $-1$ to $+1$) makes it slightly easier to optimize than sigmoid in practice, since the gradients with respect to downstream weights are not all positive or all negative for a given input.

The ReLU function $\text{ReLU}(z) = \max(0, z)$ has derivative:

$$\text{ReLU}'(z) = \begin{cases} 1 & \text{if } z > 0 \\ 0 & \text{if } z \leq 0 \end{cases}$$

For positive inputs, ReLU passes the gradient through unchanged (gradient is 1). For negative inputs, ReLU blocks the gradient entirely (gradient is 0). This has two important consequences:

1. No vanishing gradient for active neurons: When a neuron is active (output > 0), the gradient passes through without attenuation. A 100-layer ReLU network can potentially propagate gradients all the way back to the first layer without shrinkage, as long as enough neurons remain active.
2. Dying ReLU problem: If a neuron consistently outputs negative values (typically due to a large negative bias or unfortunate weight initialization), it permanently stops learning because all gradients through it are zero. This can be detected during training by monitoring the fraction of neurons with zero activations, and mitigated with careful initialization or variants like Leaky ReLU.

The ReLU gradient's binary nature means it is not just non-vanishing for positive inputs: it also cannot explode. A gradient of 1 times 1 is still 1. This stability is a major practical advantage, and it explains why the shift from sigmoid to ReLU activations in the early 2010s dramatically improved the trainability of deep networks.

The GELU function (used in modern transformers) has a smooth, data-dependent gradient that avoids both sharp saturation and hard zeros. Its derivative involves the standard normal PDF and CDF, creating a smoother gradient landscape than ReLU. GELU's derivative is positive even for moderately negative inputs, which means neurons rarely die completely, though they do attenuate gradients in the negative regime.