Activation Functions: From Sigmoid to GELU and Beyond

Neural networks derive their power from non-linearity. Without activation functions, even the deepest network would collapse into a simple linear transformation, no matter how many layers you stack. Activation functions are the secret ingredient that allows neural networks to learn complex, non-linear patterns in data.

In this chapter, we explore the evolution of activation functions, from the biologically-inspired sigmoid to modern innovations like GELU and Swish. You will understand not just what each function computes, but why it was designed that way and when to use it.

Consider stacking two linear transformations. If the first layer computes $\mathbf{h} = \mathbf{W}_1 \mathbf{x} + \mathbf{b}_1$ and the second computes $\mathbf{y} = \mathbf{W}_2 \mathbf{h} + \mathbf{b}_2$, we can substitute and simplify:

where:

• $\mathbf{x}$: the input vector
• $\mathbf{W}_1, \mathbf{W}_2$: weight matrices for layers 1 and 2
• $\mathbf{b}_1, \mathbf{b}_2$: bias vectors for layers 1 and 2
• $\mathbf{h}$: the hidden representation after the first layer
• $\mathbf{y}$: the final output

The result is equivalent to a single linear layer with weight matrix $\mathbf{W}' = \mathbf{W}_2 \mathbf{W}_1$ and bias $\mathbf{b}' = \mathbf{W}_2 \mathbf{b}_1 + \mathbf{b}_2$. No matter how many layers we add, without non-linearity, the network can only learn linear decision boundaries.

Activation functions break this limitation. By applying a non-linear function after each linear transformation, we enable networks to approximate arbitrarily complex functions.

The sigmoid function was one of the first activation functions used in neural networks. Its S-shaped curve smoothly maps any real number to a value between 0 and 1, making it interpretable as a probability.

The sigmoid function is defined as:

where:

• $z$: the input value (pre-activation), which can be any real number
• $e$: Euler's number (approximately 2.718)
• $\sigma(z)$: the output, constrained to the range $(0, 1)$

The function works by exponentiating the negative input. When $z$ is large and positive, $e^{-z}$ approaches 0, so $\sigma(z) \approx 1$. When $z$ is large and negative, $e^{-z}$ becomes very large, pushing $\sigma(z)$ toward 0. At $z = 0$, we get $\sigma(0) = 0.5$.

The sigmoid has an elegant derivative that can be expressed in terms of itself:

where:

• $\sigma'(z)$: the derivative of sigmoid with respect to $z$
• $\sigma(z)$: the sigmoid function evaluated at $z$

This property makes gradient computation efficient. Once we compute the forward pass value $\sigma(z)$, we can immediately compute the gradient without re-evaluating the exponential.

The maximum value of the derivative occurs at $z = 0$, where $\sigma'(0) = 0.25$. This means the gradient is at most 0.25, and it rapidly approaches 0 as $|z|$ increases. This phenomenon is called saturation.

Let's visualize the sigmoid function and its derivative to understand saturation:

The shaded regions highlight where saturation occurs. In these zones, the gradient is nearly zero, making learning extremely slow or impossible.

The hyperbolic tangent function addresses one limitation of sigmoid: it is zero-centered, meaning its outputs are symmetric around zero. This property helps with gradient flow during training.

where:

• $z$: the input value
• $e^z, e^{-z}$: exponentials of the positive and negative input
• $\tanh(z)$: the output, constrained to the range $(-1, 1)$

An equivalent formulation relates tanh to sigmoid:

This shows that tanh is essentially a rescaled and shifted version of sigmoid, mapping to $(-1, 1)$ instead of $(0, 1)$.

where:

• $\tanh'(z)$: the derivative of tanh with respect to $z$
• $\tanh^2(z)$: the square of the tanh output

At $z = 0$, the derivative equals 1, which is four times larger than sigmoid's maximum gradient of 0.25. This stronger gradient signal helps mitigate, though not eliminate, the vanishing gradient problem.

Both sigmoid and tanh suffer from saturation for large $|z|$ values. However, tanh's zero-centered output makes it preferable for hidden layers, while sigmoid remains useful for output layers when you need probabilities.

ReLU revolutionized deep learning. Its simplicity, computational efficiency, and resistance to vanishing gradients made training deep networks practical for the first time.

where:

• $z$: the input value
• $\max(0, z)$: returns $z$ if positive, otherwise 0

The function is piecewise linear: it passes positive values unchanged and clips negative values to zero.

At exactly $z = 0$, the derivative is technically undefined, but in practice, we typically set it to 0 or 1.

The key insight is that for positive inputs, the gradient is exactly 1. This means gradients flow through ReLU layers without diminishing, solving the vanishing gradient problem that plagued sigmoid and tanh.

While ReLU solves vanishing gradients, it introduces a new issue: neurons can "die" during training.

This typically happens when:

1. The learning rate is too high, causing large weight updates
2. Poor weight initialization pushes many neurons into negative territory
3. Strong negative gradients shift the bias terms too far negative

Leaky ReLU addresses the dying ReLU problem by allowing a small, non-zero gradient when the input is negative.

where:

• $z$: the input value
• $\alpha$: a small positive constant, typically 0.01
• $\alpha z$: the scaled negative input, giving a small but non-zero output

The function can also be written compactly as:

PReLU takes this further by making $\alpha$ a learnable parameter:

The key difference is that $\alpha$ is learned during training via backpropagation, allowing the network to determine the optimal slope for negative inputs.

For both Leaky ReLU and PReLU:

The small but non-zero gradient $\alpha$ for negative inputs allows the neuron to continue learning even when it receives negative pre-activations.

ELU provides smooth, differentiable transitions at zero and pushes mean activations closer to zero, which can accelerate learning.

where:

• $z$: the input value
• $\alpha$: a hyperparameter controlling the saturation value for negative inputs (typically 1.0)
• $e^z - 1$: an exponential term that approaches $-1$ as $z \to -\infty$

For large negative inputs, ELU saturates at $-\alpha$. This saturation provides noise robustness by limiting the influence of large negative values.

Note that for $z \leq 0$, the derivative can be computed from the function value itself, similar to sigmoid's self-referential gradient.

SELU is a self-normalizing variant of ELU. When used with proper weight initialization, SELU activations automatically converge to zero mean and unit variance, eliminating the need for batch normalization.

where the specific values are:

• $\lambda \approx 1.0507$
• $\alpha \approx 1.6733$

These constants were derived analytically to ensure self-normalizing properties.

GELU has become the default activation function in transformer architectures like BERT and GPT. Unlike ReLU, which deterministically zeroes out negative values, GELU applies a smooth, probabilistic gating.

GELU can be understood through a stochastic regularization lens. Imagine each neuron is randomly multiplied by either 0 or 1, where the probability of being "on" depends on how positive the input is. GELU computes the expected value of this stochastic process:

where $Z \sim \mathcal{N}(0, 1)$ is a standard normal random variable.

The formal definition uses the cumulative distribution function (CDF) of the standard normal distribution:

where:

• $z$: the input value
• $\Phi(z)$: the CDF of the standard normal distribution, i.e., $\Phi(z) = P(Z \leq z)$
• $z \cdot \Phi(z)$: the input scaled by the probability that a standard normal is less than $z$

The CDF $\Phi(z)$ is defined as:

where $\text{erf}$ is the error function.

Computing the error function can be expensive. A commonly used approximation is:

This approximation is accurate to within a few percent and is computationally efficient.

The derivative of GELU involves both the CDF and the probability density function (PDF) of the standard normal:

where:

• $\Phi(z)$: the standard normal CDF
• $\phi(z) = \frac{1}{\sqrt{2\pi}}e^{-z^2/2}$: the standard normal PDF

This derivative is always positive for large positive $z$, and smoothly transitions through zero.

The key difference is visible near zero: GELU smoothly curves through the origin, allowing small negative values to pass through with reduced magnitude. This smooth gating is believed to help with gradient flow and model expressiveness.

Swish and Mish are modern activation functions discovered through automated search and designed to combine the benefits of ReLU-like behavior with smooth gradients.

Swish was discovered by Google researchers using automated search techniques:

where:

• $z$: the input value
• $\sigma$: the sigmoid function
• $\beta$: a learnable or fixed parameter (often $\beta = 1$)
• $z \cdot \sigma(\beta z)$: the input scaled by the sigmoid of itself

When $\beta = 1$, this simplifies to:

Swish is non-monotonic: it dips below zero for negative inputs before rising again. This allows small negative gradients to flow backward, potentially helping escape local minima.

Mish uses the softplus function and tanh for an even smoother curve:

where:

• $\text{softplus}(z) = \ln(1 + e^z)$: a smooth approximation to ReLU
• $\tanh$: the hyperbolic tangent function

Mish has continuous derivatives of all orders, making it particularly smooth. Like Swish, it is non-monotonic with a small negative region.

Let's visualize all the activation functions we've covered in a single comprehensive comparison. This 2×4 grid allows direct comparison of each function's behavior, particularly around zero where their differences are most apparent.

Let's implement and visualize these activation functions using PyTorch. Modern deep learning frameworks provide optimized implementations of all common activations.

The output shows each activation's behavior at three key points: a negative value, zero, and a positive value. Notice how ReLU completely zeros out negative inputs, while Leaky ReLU preserves a scaled version. GELU and Swish (SiLU) show smooth, non-zero responses even for negative inputs.

You can also implement custom activations as simple functions:

The tanh approximation closely matches the exact GELU computation while being computationally efficient. The small differences are negligible in practice.

Selecting the right activation function depends on your architecture, task, and computational constraints. Here is a practical guide based on current best practices.

GELU has become the de facto standard for transformers. Its smooth gating properties align well with the attention mechanism's probabilistic nature. Major models like BERT, GPT, and their variants all use GELU.

Tanh remains common for recurrent connections because:

1. Zero-centered outputs help maintain stable hidden states
2. The bounded range prevents activations from exploding
3. Gate mechanisms (in LSTMs/GRUs) specifically rely on sigmoid's (0, 1) output

When implementing activation functions in production, consider these factors:

Computational Cost: ReLU is the fastest, requiring only a comparison and conditional. Exponential-based functions (sigmoid, tanh, ELU, SELU) are more expensive. GELU requires either the expensive error function or a polynomial approximation.

Numerical Stability: Sigmoid and softmax can overflow for very large inputs. Use numerically stable implementations that subtract the maximum value before exponentiating.

Initialization Compatibility: Some activations work best with specific initialization schemes. He initialization pairs well with ReLU variants, while Xavier/Glorot initialization suits tanh and sigmoid.

Gradient Flow: For very deep networks, favor activations with gradients that neither vanish nor explode. ReLU and its variants generally provide good gradient flow, while sigmoid and tanh can cause vanishing gradients.

Activation functions are the non-linear transformations that give neural networks their expressive power. The field has evolved from biologically-inspired functions like sigmoid to modern innovations designed for specific architectural needs.

Key takeaways:

• Sigmoid and tanh were early standards but suffer from vanishing gradients in deep networks. Sigmoid remains useful for output layers when probabilities are needed.

• ReLU revolutionized deep learning with its simple, sparse, and gradient-preserving properties. The dying ReLU problem led to variants like Leaky ReLU and ELU.

• GELU provides smooth, probabilistic gating and has become the standard for transformer architectures. Its stochastic interpretation aligns well with the attention mechanism.

• Swish and Mish offer smooth, non-monotonic alternatives that can outperform ReLU on certain tasks while maintaining good gradient flow.

• For practical applications: Start with ReLU for general feedforward networks, GELU for transformers, and task-appropriate functions for output layers.

The choice of activation function can significantly impact training dynamics and final performance. While the differences may seem subtle, they compound across millions of neurons and billions of parameters in modern networks.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about activation functions in neural networks.