Activation Functions › The Saturation Problem

Before examining individual activation functions, it helps to understand the core failure mode they all share to some degree: saturation. A function saturates when its derivative approaches zero for large input magnitudes. When that happens, the gradient that backpropagation sends through that neuron also approaches zero, and the weights feeding into that neuron stop updating. Saturated neurons are effectively frozen during training, which slows convergence dramatically.

The saturation problem becomes catastrophic in deep networks. Consider a network with $N$ layers, and suppose every activation function has a maximum gradient of $g_{\max}$. During backpropagation, gradients are multiplied together as they propagate through each layer. By the time the gradient reaches layer 1 from layer $N$, it has been multiplied by at most $g_{\max}^N$. If $g_{\max} < 1$ (which is the case for sigmoid), this product shrinks exponentially with depth. With just 10 layers, a gradient that starts at 1.0 at the output arrives at the first layer as $(g_{\max})^{10}$. For sigmoid, where $g_{\max} = 0.25$, this gives $(0.25)^{10} \approx 10^{-6}$: the gradient has been reduced by a factor of one million. The first few layers learn almost nothing.

This phenomenon, called the vanishing gradient problem, was one of the central obstacles to training deep networks throughout the 1990s and 2000s. Understanding activation functions through the lens of their gradient behavior is the most important lens available, because gradient flow is what determines whether training works at all.

The sigmoid function, also called the logistic function, was the dominant activation in neural networks from the 1980s through the early 2010s. It maps any real-valued input to a value between 0 and 1, which made it appealing for early work on binary classification and probabilistic outputs.

The sigmoid function is defined as:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

where:

• $z$: the pre-activation value (the weighted sum $\mathbf{w}^T \mathbf{x} + b$)
• $e^{-z}$: the exponential of the negated input, which ensures the function maps to $(0, 1)$

The derivative of sigmoid has a clean form. Letting $\sigma = \sigma(z)$:

$$\frac{d\sigma}{dz} = \sigma(1 - \sigma)$$

This is why sigmoid seemed mathematically elegant: its own value tells you its derivative. But the derivative has a critical problem. The maximum value of $\sigma(1 - \sigma)$ occurs at $z = 0$ where $\sigma = 0.5$, giving $0.5 \times 0.5 = 0.25$. For $|z| > 3$, the derivative is already below 0.05. For $|z| > 6$, it is below $0.002$.

When inputs to a sigmoid neuron are large in magnitude (either very positive or very negative), the gradient is nearly zero. During backpropagation, this small gradient multiplies the gradients flowing back through deeper layers, causing the vanishing gradient problem that we cover in depth in the chapter on Vanishing Gradients. Sigmoid networks with many layers effectively fail to train their early layers, which is why deep sigmoid networks were considered impractical for most of the 1990s.

A second issue with sigmoid is that it is not zero-centered. Its output range is $(0, 1)$, meaning the average output is positive. This causes the gradients for weights in a layer to always have the same sign (either all positive or all negative), because the gradient of the loss with respect to a weight $w_i$ in a layer is $\delta \cdot x_i$, where $x_i$ is the input to that layer. If all inputs $x_i$ are positive (because the previous layer used sigmoid), then the gradient always has the same sign as $\delta$. This means weight updates in a given layer must all move in the same direction, causing the optimizer to zigzag through the loss landscape rather than heading directly toward the optimum. This effect is subtle but real, and it was one motivation for developing zero-centered alternatives.

The hyperbolic tangent function, commonly written tanh, was proposed as an improvement over sigmoid to address the zero-centering issue. It was the dominant activation function in recurrent networks and deep feedforward networks from the late 1990s through the mid-2010s.

Tanh is defined as:

$$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$$

where:

• $e^z$ and $e^{-z}$: the forward and backward exponentials of the input

Notice that tanh is simply a rescaled and shifted version of sigmoid:

$$\tanh(z) = 2\sigma(2z) - 1$$

This algebraic relationship means tanh inherits the same S-shaped curve as sigmoid but stretches it vertically to span $(-1, 1)$ instead of $(0, 1)$.

Tanh maps inputs to the range $(-1, 1)$, making it zero-centered. Its derivative is:

$$\frac{d\,\tanh(z)}{dz} = 1 - \tanh^2(z)$$

At $z = 0$, the derivative is 1, which is four times stronger than sigmoid's maximum of 0.25. This means gradients flow much more easily near the origin. However, tanh still saturates: for $|z| > 2$, its derivative falls below 0.07, and for $|z| > 3$ it is nearly zero. Tanh trades reduced saturation range (saturation kicks in at $|z| > 2$ rather than $|z| > 1$) for zero-centering.

In practice, tanh outperforms sigmoid in hidden layers because of zero-centering and a stronger gradient near the origin. It is still used in LSTM gate computations and GRU architectures, where the bounded range $(-1, 1)$ has a useful interpretation as a scaled-memory value. The cell state in an LSTM is bounded by the tanh function applied to the candidate activation, which prevents unbounded growth in the memory. Sigmoid, by contrast, remains appropriate for the output layer of binary classifiers because you need a value in $(0, 1)$ to interpret as a probability.

The figure below shows sigmoid and tanh side by side, including their derivatives, making the saturation behavior visually clear.