Activation Functions for Neural Networks

In the previous chapter on Linear Classifiers, we saw how a network layer computes a weighted sum of its inputs and passes the result through a softmax function to produce class probabilities. That operation is inherently linear: no matter how many such layers you stack, the composition of linear functions is still a linear function. A 100-layer network without anything nonlinear between the layers is mathematically identical to a single matrix multiplication. Activation functions break this constraint. They are the nonlinear gates placed after each linear transformation, and without them deep networks would have no more expressive power than a single-layer perceptron.

To appreciate why this matters, consider what a linear layer does. A linear transformation maps the input space to an output space through rotation, scaling, and reflection. You can stack as many of these transformations as you like, but the final result is still just another rotation, scaling, and reflection of the original input. There is no way to separate points that require a curved boundary, no way to model interactions between features that are not already captured by a single weighted sum. The moment you insert a nonlinear activation function between two linear layers, everything changes. Now the network can represent curved decision boundaries, and as you add more layers with more nonlinear activations, the representable function class grows exponentially.

Choosing the right activation function is not a cosmetic decision. The function you pick determines how gradients flow backward through the network during training, how quickly weights converge, whether certain neurons become permanently inactive, and how the network behaves in deep architectures. This chapter covers every major activation function used in modern neural networks: from the classical sigmoid and tanh through ReLU and its variants, through the exponential families ELU and SELU, all the way to the smooth modern functions like GELU, Swish, and Mish that power today's language models.

Before examining individual activation functions, it helps to understand the core failure mode they all share to some degree: saturation. A function saturates when its derivative approaches zero for large input magnitudes. When that happens, the gradient that backpropagation sends through that neuron also approaches zero, and the weights feeding into that neuron stop updating. Saturated neurons are effectively frozen during training, which slows convergence dramatically.

The saturation problem becomes catastrophic in deep networks. Consider a network with $N$ layers, and suppose every activation function has a maximum gradient of $g_{\max}$. During backpropagation, gradients are multiplied together as they propagate through each layer. By the time the gradient reaches layer 1 from layer $N$, it has been multiplied by at most $g_{\max}^N$. If $g_{\max} < 1$ (which is the case for sigmoid), this product shrinks exponentially with depth. With just 10 layers, a gradient that starts at 1.0 at the output arrives at the first layer as $(g_{\max})^{10}$. For sigmoid, where $g_{\max} = 0.25$, this gives $(0.25)^{10} \approx 10^{-6}$: the gradient has been reduced by a factor of one million. The first few layers learn almost nothing.

This phenomenon, called the vanishing gradient problem, was one of the central obstacles to training deep networks throughout the 1990s and 2000s. Understanding activation functions through the lens of their gradient behavior is the most important lens available, because gradient flow is what determines whether training works at all.

The sigmoid function, also called the logistic function, was the dominant activation in neural networks from the 1980s through the early 2010s. It maps any real-valued input to a value between 0 and 1, which made it appealing for early work on binary classification and probabilistic outputs.

The sigmoid function is defined as:

where:

• $z$: the pre-activation value (the weighted sum $\mathbf{w}^T \mathbf{x} + b$)
• $e^{-z}$: the exponential of the negated input, which ensures the function maps to $(0, 1)$

The derivative of sigmoid has a clean form. Letting $\sigma = \sigma(z)$:

This is why sigmoid seemed mathematically elegant: its own value tells you its derivative. But the derivative has a critical problem. The maximum value of $\sigma(1 - \sigma)$ occurs at $z = 0$ where $\sigma = 0.5$, giving $0.5 \times 0.5 = 0.25$. For $|z| > 3$, the derivative is already below 0.05. For $|z| > 6$, it is below $0.002$.

When inputs to a sigmoid neuron are large in magnitude (either very positive or very negative), the gradient is nearly zero. During backpropagation, this small gradient multiplies the gradients flowing back through deeper layers, causing the vanishing gradient problem that we cover in depth in the chapter on Vanishing Gradients. Sigmoid networks with many layers effectively fail to train their early layers, which is why deep sigmoid networks were considered impractical for most of the 1990s.

A second issue with sigmoid is that it is not zero-centered. Its output range is $(0, 1)$, meaning the average output is positive. This causes the gradients for weights in a layer to always have the same sign (either all positive or all negative), because the gradient of the loss with respect to a weight $w_i$ in a layer is $\delta \cdot x_i$, where $x_i$ is the input to that layer. If all inputs $x_i$ are positive (because the previous layer used sigmoid), then the gradient always has the same sign as $\delta$. This means weight updates in a given layer must all move in the same direction, causing the optimizer to zigzag through the loss landscape rather than heading directly toward the optimum. This effect is subtle but real, and it was one motivation for developing zero-centered alternatives.

The hyperbolic tangent function, commonly written tanh, was proposed as an improvement over sigmoid to address the zero-centering issue. It was the dominant activation function in recurrent networks and deep feedforward networks from the late 1990s through the mid-2010s.

Tanh is defined as:

where:

• $e^z$ and $e^{-z}$: the forward and backward exponentials of the input

Notice that tanh is simply a rescaled and shifted version of sigmoid:

This algebraic relationship means tanh inherits the same S-shaped curve as sigmoid but stretches it vertically to span $(-1, 1)$ instead of $(0, 1)$.

Tanh maps inputs to the range $(-1, 1)$, making it zero-centered. Its derivative is:

At $z = 0$, the derivative is 1, which is four times stronger than sigmoid's maximum of 0.25. This means gradients flow much more easily near the origin. However, tanh still saturates: for $|z| > 2$, its derivative falls below 0.07, and for $|z| > 3$ it is nearly zero. Tanh trades reduced saturation range (saturation kicks in at $|z| > 2$ rather than $|z| > 1$) for zero-centering.

In practice, tanh outperforms sigmoid in hidden layers because of zero-centering and a stronger gradient near the origin. It is still used in LSTM gate computations and GRU architectures, where the bounded range $(-1, 1)$ has a useful interpretation as a scaled-memory value. The cell state in an LSTM is bounded by the tanh function applied to the candidate activation, which prevents unbounded growth in the memory. Sigmoid, by contrast, remains appropriate for the output layer of binary classifiers because you need a value in $(0, 1)$ to interpret as a probability.

The figure below shows sigmoid and tanh side by side, including their derivatives, making the saturation behavior visually clear.

The Rectified Linear Unit (ReLU) was proposed as a simple solution to saturation. By making the function piecewise linear, it guarantees that the gradient on the positive side is always exactly 1, no matter how large the input. This is the key reason deep networks became trainable in practice.

ReLU is defined as:

where:

• $z$: the pre-activation value
• $\max(0, z)$: takes the larger of 0 and $z$, clipping negative values to zero

The derivative is straightforward:

At $z = 0$, the derivative is technically undefined, but in practice implementations return 0 or 1 here without issue.

The gradient being 1 for all positive inputs eliminates saturation on the positive side entirely. When a neuron is active (positive input), gradients flow through it unchanged. This makes training deep networks with ReLU dramatically faster than with sigmoid or tanh. ReLU was a key enabler of the deep learning revolution in the early 2010s, particularly for convolutional networks like AlexNet (2012) and VGG (2014), and it remains the default activation for most hidden layers in feedforward and convolutional networks today.

ReLU also has a property called sparsity induction: when many neurons have negative pre-activation inputs, they output exactly zero. For a typical network with random weights early in training, roughly half the neurons are inactive on any given input. This sparsity means fewer neurons participate in each forward pass, which has two practical benefits. First, it reduces co-adaptation between neurons: neurons cannot rely on the output of another neuron that is often inactive, which acts as an implicit regularizer. Second, sparse activations are computationally efficient, particularly for hardware accelerators that can skip zero-valued computations.

There is also a biological motivation for ReLU-like functions. Neurons in the brain have a firing threshold: below a certain stimulus level, they produce no response. Above it, they fire at a rate proportional to the stimulus. This is qualitatively similar to ReLU, though the analogy should not be taken too literally.

ReLU's left side is its weakness. Any neuron that receives a negative pre-activation outputs zero and contributes nothing to the forward pass. More critically, its gradient is also zero, so no update flows back through it. If a neuron consistently receives negative inputs across the entire training set, its weights stop updating entirely. The neuron is dead.

This happens in practice more often than expected. A large learning rate can push the weights in one training step such that a neuron's pre-activation becomes negative for all training examples. Once that happens, the neuron's gradient is zero for all examples, and its weights receive no signal to move back. The dead state is an absorbing state: once a neuron dies, it cannot recover on its own.

Networks trained with ReLU can have 10-40% of neurons permanently dead, depending on architecture, learning rate, and weight initialization. With a learning rate that is too high, a large weight update can drive many neurons negative simultaneously. With poor initialization (weights that are too large), the same effect can happen at the start of training. The dying ReLU problem is most severe in early layers because those neurons receive less direct signal from the loss.

Several practical measures reduce the risk:

• Use a small learning rate, especially early in training
• Initialize weights with He initialization ($\text{variance} = 2/n$, where $n$ is the fan-in), which keeps pre-activations well-distributed around zero
• Use gradient clipping to prevent large weight updates
• Monitor the fraction of dead neurons during training as a diagnostic

The fix at the architectural level is to ensure that at least some gradient can flow even when $z < 0$. This led to several ReLU variants.

Both Leaky ReLU and Parametric ReLU (PReLU) address the dying neuron problem by ensuring that negative pre-activations still produce a nonzero gradient during backpropagation. They differ in whether the negative-side slope is fixed or learned.

Leaky ReLU introduces a small slope on the negative side, so the gradient never completely vanishes:

where:

• $z$: the pre-activation value
• $\alpha$: a small positive constant controlling the negative slope, typically $\alpha = 0.01$

The derivative is:

With $\alpha = 0.01$, the negative slope sends a gradient of 0.01 back through a dead neuron on every backward pass. The neuron can now gradually revive as its weights receive small but nonzero updates. In practice, Leaky ReLU often trains slightly faster than ReLU and effectively eliminates permanently dead neurons. The tradeoff is a small loss of the sparsity property: neurons no longer output exactly zero for negative inputs, so the regularizing effect of sparsity is reduced.

The choice of $\alpha = 0.01$ is a heuristic. Values between 0.01 and 0.1 are commonly used. Larger $\alpha$ values improve gradient flow more aggressively but at the cost of less sparsity and a less clean identity-like behavior on the positive side. Values above 0.3 start to resemble a linear function more than a rectifier.

Parametric ReLU (PReLU) takes this further by making $\alpha$ a learnable parameter:

where $\alpha$ is now a learned parameter updated by gradient descent alongside the other network weights. Each neuron can have its own $\alpha$, or a single $\alpha$ can be shared across all neurons in a layer.

The gradient with respect to $\alpha$ for a neuron with negative input $z < 0$ is:

where:

• $L$: the loss function
• $a$: the neuron's output
• $z$: the pre-activation value (the input to the activation)

PReLU lets the network learn the optimal negative slope for each context. If the data does not benefit from negative activations, the learned $\alpha$ will approach zero, recovering ReLU behavior. If negative activations help, the network will use them. PReLU was introduced in the context of deep residual networks and showed small but consistent improvements over fixed Leaky ReLU in image recognition tasks. The extra parameters introduced by per-neuron $\alpha$ values are typically negligible compared to the weight matrices, so the parameter overhead is minimal.

A related variant, Randomized Leaky ReLU (RReLU), samples $\alpha$ uniformly from a fixed interval during training, then uses a fixed value (the interval's mean) at test time. The stochastic negative slope acts as a regularizer. This provides some of the same benefit as dropout without zeroing entire neurons. RReLU has shown small gains in image recognition benchmarks and is a useful technique to know, though it is less commonly used than Leaky ReLU or PReLU in production architectures.

The following plots show how ReLU, Leaky ReLU, and ELU compare in both their function values and gradients:

Exponential Linear Units address a different problem from the dying neuron issue. Even when ReLU neurons are alive, the average activation of a ReLU layer is always positive (since ReLU clips negatives to zero). This causes the same bias-shift problem that sigmoid suffers from: if all outputs from a layer are positive, the gradients for all weights in the next layer must have the same sign, leading to zigzag optimization paths. ELU restores a mean closer to zero by giving the negative side a smooth exponential shape rather than a hard zero.

ELU is defined as:

where:

• $z$: the pre-activation value
• $\alpha$: a positive hyperparameter (typically $\alpha = 1$) controlling the saturation point for negative inputs
• $e^z$: exponential of the pre-activation, which smoothly approaches $-\alpha$ as $z \to -\infty$

The derivative is:

For large negative inputs, ELU approaches $-\alpha$, so the mean output of an ELU layer can be zero if the distribution of pre-activations is approximately symmetric around zero. This zero-mean property means the gradients for weights feeding into the next layer do not suffer from the systematic sign bias that ReLU creates. ELU is more computationally expensive than ReLU because of the exponential computation, but in architectures where zero-centering matters (particularly deep feedforward networks trained without batch normalization) it often converges faster.

The key difference between ELU and Leaky ReLU is the shape of the negative side. Leaky ReLU uses a linear negative side (a straight line with slope $\alpha$). ELU uses an exponential that starts at 0 when $z = 0$ and saturates at $-\alpha$ as $z \to -\infty$. The saturation means ELU is more less brittle to very negative inputs than Leaky ReLU, which grows without bound in the negative direction. This bounded negative side is important for mean stability.

Scaled Exponential Linear Unit (SELU) takes ELU further with a specific pair of scaling constants $\lambda$ and $\alpha$ chosen to enforce self-normalizing behavior:

where:

• $\lambda \approx 1.0507$: the outer scaling constant
• $\alpha \approx 1.6733$: the negative saturation constant

These specific values were derived analytically by Klambauer et al. (2017) to ensure that if the inputs to a SELU layer are approximately standardized (mean 0, variance 1), the outputs will also be approximately standardized, regardless of network depth. This is the self-normalizing property: SELU networks tend to maintain stable activations throughout training without explicit batch normalization.

The self-normalization property is remarkable because it removes the need for one of the most important regularization techniques in deep learning. Batch normalization is computationally expensive, requires careful tuning, and interacts poorly with small batch sizes and certain architectures. SELU achieves similar statistical guarantees through the activation function itself.

SELU works best under specific conditions:

• Weights initialized with LeCun normal initialization ($\text{mean} = 0$, $\text{variance} = 1/n$, where $n$ is the number of input units)
• No batch normalization or dropout (which would break the statistical properties SELU relies on)
• Fully connected layers (convolutional layers need extra care because the spatial structure creates dependencies that violate SELU's assumptions)
• Data that is approximately standardized before entering the network

When conditions are met, SELU can train deep fully connected networks reliably without normalization layers. In practice, it is used less often than ReLU because transformers and convolutional architectures use different normalization strategies, but it remains relevant for structured and tabular deep learning where batch normalization interactions are problematic or where small batch sizes make batch statistics unreliable.

GELU is the activation function used in BERT, GPT, and most modern transformer architectures. Understanding why it works requires examining its mechanism, which is more subtle than it might appear.

GELU combines the identity function (like ReLU's positive side) with a smooth, stochastic gating mechanism. The exact definition is:

where:

• $z$: the pre-activation value
• $\Phi(z)$: the cumulative distribution function (CDF) of the standard normal distribution, $\Phi(z) = P(X \leq z)$ for $X \sim \mathcal{N}(0, 1)$

The intuition is that GELU scales the input $z$ by the probability that a standard normal sample would be less than $z$. When $z$ is very large and positive, $\Phi(z) \approx 1$, so GELU acts like the identity. When $z$ is very large and negative, $\Phi(z) \approx 0$, so the output is nearly zero. Near $z = 0$, the gate is approximately 0.5, giving a smooth transition.

This is equivalent to saying: the neuron stochastically decides whether to pass its input or zero it out, where the probability of passing is $\Phi(z)$. Neurons with larger positive activations are more likely to be "on," and neurons with negative activations are more likely to be "off." The difference from ReLU is that the transition is smooth rather than hard, which allows gradients to flow through even slightly negative activations. Compared to a hard binary gate (as in ReLU), the smooth probabilistic gate provides more information to the optimizer about how the input contributed to the output, which translates to better gradient signal during backpropagation.

Another way to see the GELU intuition is as a regularized identity. A standard linear function $f(z) = z$ is the simplest possible activation: it passes everything through. GELU is like a linear function multiplied by a confidence score: the more strongly positive this activation is (based on its relationship to the distribution of all activations), the more of it we pass through.

The exact GELU requires computing the standard normal CDF, which involves the error function:

where:

• $\text{erf}(x) = \frac{2}{\sqrt{\pi}} \int_0^x e^{-t^2} dt$: the error function

This is expensive to compute exactly. A fast approximation using tanh is commonly used in practice:

where:

• $0.044715$: an empirical correction constant
• $\sqrt{2/\pi} \approx 0.7978$: the normalization factor for the approximation

This approximation is accurate to within a few percent across the range that matters for neural network training and is significantly faster than computing the true error function. PyTorch provides both variants: nn.GELU() uses the exact form by default, while nn.GELU(approximate='tanh') uses the faster approximation. The original BERT paper used the tanh approximation; later work and most current implementations use the exact version.

GELU has several properties that make it well-suited to transformers.

First, the smooth gradient everywhere (no hard zero at $z = 0$) helps with the optimization landscape in deep architectures with many layers. Transformers stack dozens of identical blocks, and any activation function that creates hard discontinuities in the gradient can cause instability as gradients multiply through all those layers.

Second, GELU's slight dip into negative territory near $z \approx -0.17$ before recovering to zero produces a non-monotone shape. Unlike ReLU, which is monotone everywhere, GELU can produce slightly negative outputs even for slightly negative inputs. This non-monotone property empirically helps the network learn more complex feature representations by allowing the activation to express subtle distinctions in the negative region.

Third, the stochastic interpretation aligns well with dropout-like regularization. GELU can be understood as a smooth approximation to a neuron that randomly drops its activation based on a Bernoulli distribution parameterized by $\Phi(z)$. This provides an implicit regularization effect similar to dropout, without the discreteness and variance that dropout introduces.

BERT, GPT-2, GPT-3, and most subsequent transformer architectures use GELU or a variant. The choice has become canonical for transformers in the same way that ReLU became canonical for CNNs.

Swish (also called SiLU, for Sigmoid Linear Unit) was discovered by automated neural architecture search at Google in 2017. It resembles GELU but is simpler in formulation and has become the activation function of choice in several important vision architectures.

Swish is defined as:

where:

• $z$: the pre-activation value
• $\sigma(z) = 1/(1 + e^{-z})$: the sigmoid function used as a soft gate

Like GELU, Swish scales the input by a smooth gate function. The difference is that Swish uses sigmoid as the gate rather than the normal CDF. For large positive $z$, $\sigma(z) \to 1$ so Swish approaches the identity. For large negative $z$, $\sigma(z) \to 0$ so Swish approaches zero. The function is non-monotone with a minimum around $z \approx -1.28$ where the output is approximately $-0.28$.

The derivative of Swish is:

where:

• $\sigma(z)$: the sigmoid gate value at $z$
• $z(1 - \sigma(z))$: the product contribution from the input-dependent component

This derivative is smooth everywhere and never hard-zeros the gradient, which aids gradient flow in deep networks.

Swish and GELU are closely related and perform similarly in practice. The main differences are:

• Swish uses sigmoid as the gate function; GELU uses the normal CDF. Both give similar values in the range that matters for neural activations, because sigmoid and the normal CDF have similar shapes.
• Swish has a learnable variant where the sigmoid is scaled by a learnable parameter $\beta$: $z \cdot \sigma(\beta z)$. When $\beta = 1$, this is standard Swish. When $\beta \to 0$, it approaches a linear function. When $\beta \to \infty$, the sigmoid sharpens into a step function and Swish approaches ReLU. This parameterization allows the network to learn the appropriate gate sharpness for each layer.
• GELU is used more in NLP (following BERT's success); Swish/SiLU is more common in vision models and EfficientNet architectures.
• For practical purposes, you can treat GELU and SiLU as interchangeable unless you have specific reasons to prefer one.

PyTorch provides both: torch.nn.GELU() and torch.nn.SiLU().

Mish, proposed by Diganta Misra in 2019, is a self-regularized non-monotone activation that has been used successfully in computer vision and some NLP applications.

Mish is defined as:

where:

• $z$: the pre-activation value
• $\text{softplus}(z) = \ln(1 + e^z)$: a smooth approximation to the ReLU function that avoids the hard zero
• $\tanh(\cdot)$: the hyperbolic tangent applied to the softplus output

Mish, like Swish and GELU, is non-monotone with a slight negative region near its minimum. Its output is unbounded above (the positive side grows like the identity) but bounded below by approximately $-0.31$. The softplus inner function smoothly transitions from near-zero for large negative inputs to near-linear for large positive inputs, and the tanh applied to it produces a gate that smoothly goes from 0 to 1. The product $z \cdot \text{gate}$ then passes increasingly more of the input as $z$ grows more positive.

The derivative of Mish:

where:

• $\text{sech}^2(\cdot)$: the derivative of tanh
• $\sigma(z)$: the sigmoid function, which is the derivative of softplus

Mish is more computationally expensive than GELU or Swish due to the compound computation involving both a softplus and a tanh. However, benchmarks show it outperforms ReLU on image classification tasks, and it has been adopted in YOLO-based object detection architectures where the small accuracy gain per layer compounds across the many layers in the network. For NLP tasks, the differences compared to GELU are small enough that GELU remains the default.

The key insight behind Mish is that applying tanh to softplus creates a particularly smooth gate: softplus is infinitely differentiable, and applying another smooth function to it produces a gate with very gentle second-order derivatives. This smoothness in the gate function translates to smoothness in the overall activation, which tends to produce better-conditioned optimization landscapes.

The next figure compares GELU, Swish/SiLU, and Mish, showing how these modern smooth activations resemble each other while differing from ReLU in the negative region:

To compare these functions, it helps to look at four properties: the output range, whether the function is zero-centered, the gradient at zero, and the computational cost.

To understand how these functions affect gradient flow in deep networks, consider what happens when you backpropagate a gradient $\delta$ through a layer with activation $f$. The gradient signal that reaches the layer below is $\delta \cdot f'(z)$. If $f'(z) < 1$ consistently, the gradient shrinks at each layer. If $f'(z) > 1$, it grows (which can cause exploding gradients).

For sigmoid, $f'(z) \leq 0.25$ everywhere. Backpropagating through $N$ sigmoid layers multiplies the gradient by at most $0.25^N$. With just 10 layers, this is $(0.25)^{10} \approx 10^{-6}$: a million-fold reduction. The early layers receive essentially no gradient and their weights barely move.

For ReLU, $f'(z) = 1$ on the active side. Backpropagating through $N$ active ReLU neurons multiplies the gradient by $1^N = 1$: no shrinkage at all. This is the core reason ReLU enabled training of deep networks. The caveat is that dead neurons (the $f'(z) = 0$ side) block gradient flow entirely, which is why the dying ReLU problem matters for deep or poorly initialized networks.

For GELU and Swish, the derivative near zero is approximately 0.5, and the gradient grows to approximately 1 for large positive inputs. These smooth functions have slightly smaller average gradients than ReLU on the positive side but handle the transition at zero more gracefully, avoiding the hard zero that causes dying neurons. The gradient for GELU varies between approximately 0.5 and 1.0 for typical pre-activation values, which means GELU networks do experience some gradient attenuation but far less than sigmoid networks.

The following plot illustrates gradient magnitude decay across layers for sigmoid versus ReLU, showing the dramatic difference that activation choice makes in deep networks:

In controlled experiments on standard benchmarks, the relative convergence speed of activation functions is roughly:

GELU $\approx$ Swish $\approx$ Mish $>$ ELU $>$ Leaky ReLU $\approx$ ReLU $>$ tanh $>$ sigmoid

This ordering is approximate and architecture-dependent. The modern smooth functions (GELU, Swish, Mish) often converge faster and to slightly better optima, particularly in deeper networks. The advantage over ReLU is often small (a few percent in final accuracy) but consistent enough across benchmarks that transformers standardized on GELU.

The reason the smooth activations tend to converge faster is subtle. ReLU creates a piecewise-linear loss landscape: the loss surface is flat in all the directions where neurons are dead. This means gradient descent must take many small steps around the flat regions. GELU and Swish produce a smoother landscape with fewer flat regions, so gradient descent can move more directly toward the optimum. The difference is not dramatic for shallow networks, but compounds over many layers.

To make gradient flow concrete, let's trace what happens to a gradient as it passes through a single neuron with different activation functions. Suppose the neuron has a pre-activation value of $z = -0.5$ (slightly negative, a common situation early in training when weights are small and random).

For each activation, the output value and gradient are:

Sigmoid ($z = -0.5$): $\sigma(-0.5) = 1/(1 + e^{0.5}) \approx 0.378$. Gradient: $0.378 \times (1 - 0.378) \approx 0.235$. The gradient has been attenuated to 23.5% of its original value.

Tanh ($z = -0.5$): $\tanh(-0.5) \approx -0.462$. Gradient: $1 - (-0.462)^2 \approx 0.787$. The gradient retains 78.7% of its original value, much better than sigmoid.

ReLU ($z = -0.5$): $\text{ReLU}(-0.5) = 0$. Gradient: $0$. The gradient is completely blocked. This neuron contributes nothing to training for this input.

Leaky ReLU ($z = -0.5$, $\alpha = 0.01$): $\text{LeakyReLU}(-0.5) = -0.005$. Gradient: $0.01$. The gradient is tiny (1% of original) but nonzero, allowing the neuron to eventually recover.

GELU ($z = -0.5$): $\text{GELU}(-0.5) \approx -0.154$. Gradient $\approx 0.345$. The gradient retains 34.5% of its original value, substantially more than ReLU's zero.

Swish ($z = -0.5$): $\text{Swish}(-0.5) = -0.5 \times \sigma(-0.5) \approx -0.189$. Gradient $\approx 0.324$. Similar to GELU.

This single-neuron comparison captures the key tradeoff: ReLU completely blocks gradient flow for negative inputs (dying neuron problem), while smooth activations like GELU and Swish preserve a substantial fraction of the gradient, enabling recovery from initial negative activations.

This section implements all major activation functions from scratch and compares convergence behavior on a classification task.

We begin by importing the necessary libraries and implementing each activation function as a pure NumPy computation. This helps build intuition before using PyTorch's built-in versions.

Each activation function is implemented as a NumPy function alongside its derivative. This makes it easy to compare their shapes and gradient properties interactively.

The implementations follow the mathematical definitions directly. Notice that each function uses np.clip to avoid numerical overflow when computing exponentials of large negative or positive values. This is essential for numerical stability: without clipping, np.exp(z) for $z < -500$ would underflow to zero, and for $z > 500$ would overflow to infinity.

Before running the full convergence comparison, let's verify our derivative implementations numerically. A correctly implemented derivative should match the numerical gradient (finite difference approximation) closely.

The max errors should all be on the order of $10^{-10}$ or smaller, confirming our derivative implementations are correct. Numerical gradient checking like this is a standard sanity check when implementing neural network components.

We build a small feedforward network in PyTorch and train it with each activation function on a synthetic binary classification task. The dataset has realistic class overlap to ensure not all activations achieve perfect accuracy.

One practical use of the activation analysis above is diagnosing dead ReLU neurons during training. We can measure the fraction of neurons with zero output across all inputs in the dataset to identify whether dying ReLU is occurring.

The key parameters to consider when using activation functions are:

• negative_slope (Leaky ReLU): The slope for negative inputs. Typical value is 0.01. Higher values increase gradient flow through negative activations but reduce sparsity.
• alpha (ELU): The saturation level for negative inputs, typically 1.0. Controls the mean activation level. Higher $\alpha$ makes the negative saturation more pronounced.
• approximate (GELU): Whether to use the tanh approximation. approximate='tanh' is faster on hardware without good erf support; approximate='none' is exact and is the default in modern PyTorch.
• inplace: Whether to compute the activation in-place to save memory. Use inplace=True for ReLU in memory-constrained settings. Note that inplace operations can cause issues with autograd in some complex architectures.

The choice of activation function depends on the architecture, the training setup, and the task. Here are practical guidelines for common scenarios, along with the reasoning behind each recommendation.

Transformer architectures (BERT, GPT, T5, etc.): Use GELU or SiLU. These are the standard choices, supported by extensive empirical evidence across language tasks. The smooth gating property and slight negative region appear to help with the complex feature interactions in self-attention layers. The choice between GELU and SiLU is largely arbitrary in practice: both perform similarly, and GELU has the advantage of historical precedent.

Convolutional networks (ResNets, EfficientNets, YOLO): ReLU is still the safe default for classification. Swish/SiLU is used in EfficientNet with consistent improvements. Mish has shown gains in YOLO object detection. The differences are typically small (1-2% accuracy), but for tasks where you are competing on benchmark leaderboards, using the right modern activation can make a measurable difference.

Recurrent networks (LSTMs, GRUs): Use tanh for the cell state update and sigmoid for the gates. These specific choices are baked into the LSTM and GRU equations and should not be changed without redesigning the gate structure. The bounded range of tanh ($(-1, 1)$) is important for preventing the cell state from growing unboundedly, and the $(0, 1)$ range of sigmoid is required for the gating interpretation where 0 means "forget everything" and 1 means "remember everything."

Fully connected networks without batch normalization: Consider ELU or SELU. ELU's zero-mean property reduces the co-adaptation problem and can improve convergence when you cannot use batch normalization (for example, with very small batch sizes or in sequential models). SELU provides self-normalization if you satisfy its initialization and architecture constraints, which can be particularly valuable when you want a deep network without any normalization layers.

Output layers: The activation function for the output depends entirely on the task:

• Binary classification: Sigmoid, to produce a probability in $(0, 1)$
• Multiclass classification: Softmax, to produce a probability distribution over classes
• Regression: No activation or identity, to allow unbounded outputs
• Bounded regression (e.g., predicting a value between 0 and 1): Sigmoid
• Multi-label classification (each output is an independent binary variable): Sigmoid applied independently to each output

Early stopping and diagnostic: If your network is not training at all (loss not decreasing after the first few batches), the most common culprits are bad learning rate, bad initialization, and dying ReLU. Switching to Leaky ReLU or GELU can quickly rule out the activation as the cause.

Activation functions play a somewhat different role in different parts of a transformer compared to a feedforward network.

In a standard feedforward network, every hidden layer has an activation function, and the depth of the network means gradients must propagate through many activation functions. The choice of activation function is critical for gradient flow.

In a transformer, the self-attention computation itself is linear (the attention weights are applied to the values through a weighted sum), and the nonlinearity is concentrated in the feed-forward sub-layers that follow each attention block. These feed-forward layers are typically two-layer MLPs: a linear layer that expands the dimension, followed by an activation function, followed by a linear layer that contracts back to the model dimension. GELU is the standard choice for this nonlinearity in transformers.

The attention computation in transformers uses softmax, not a traditional activation function, to normalize the attention weights. Softmax is not a nonlinearity in the same sense as ReLU or GELU: it is applied across the sequence dimension to produce a probability distribution, not applied element-wise to introduce nonlinearity within a neuron. The feed-forward sub-layers are where the true element-wise nonlinearity lives in a transformer.

Some newer transformer variants (including LLaMA and Mistral) use Swish/SiLU with a gated structure in the feed-forward layers. This gated variant, called SwiGLU, computes two parallel linear projections of the input and multiplies them element-wise after applying SiLU to one of them. We will cover gated linear units and SwiGLU in detail when we reach the chapter on Transformer Blocks and Feed-Forward Networks.

No activation function is perfect for all situations. Understanding the failure modes helps you debug training problems and make better architecture choices.

The dying ReLU problem remains the most common activation-related training failure. Networks trained with large learning rates, without proper weight initialization, or with many layers are most susceptible. If you observe that training loss stops improving early in training and learning rate adjustments do not help, checking the fraction of dead neurons is a useful diagnostic step. A simple approach is to run the entire training set through the model and check how many neurons output zero for all inputs, as shown in the code above. If more than 20-30% of neurons are dead, switching to Leaky ReLU or GELU is likely to help.

GELU and Swish are computationally more expensive than ReLU due to the sigmoid or error function computation. For latency-critical inference applications, this overhead can matter. On modern GPU hardware, the difference is usually negligible because the computation is memory-bandwidth-bound rather than compute-bound. However, on CPU inference or edge devices, the transcendental function evaluations (exponential, tanh) can be a real bottleneck. Profiling on your target hardware before committing to a smooth activation is good practice.

SELU's self-normalization property only holds under strict conditions. Batch normalization, dropout, or skip connections all break the statistical invariant that SELU relies on. Using SELU in a standard ResNet without removing these components is counterproductive and will not provide the self-normalization benefit. SELU is best suited to pure feedforward architectures where you have full control over the normalization strategy.

For activation functions in quantized or pruned models (where weights are represented in low precision), smooth activations like GELU can cause accuracy degradation because the slight negative values are lost when the network is quantized to integers. ReLU's hard zero is quantization-friendly: the threshold at zero is a natural fit for integer quantization schemes. If you need to quantize a model after training, testing whether switching from GELU to ReLU before quantization improves post-quantization accuracy is worth doing.

The relationship between activation functions and batch normalization is also important. Batch normalization is often applied before the activation function (in post-norm designs like the original ResNet) or after it (in some transformer variants). The position of batch normalization relative to the activation function changes the effective input distribution that the activation sees. If batch normalization is applied before the activation, the activation always sees inputs that are approximately standardized (mean 0, variance 1), which means it always operates near $z = 0$ where its gradient is at its highest. This reduces the saturation problem for sigmoid and tanh, making them more viable with batch normalization than without. Pre-norm transformers, which apply layer normalization before the attention and feed-forward sub-layers, benefit from a similar effect.

Finally, the activation function interacts with the weight initialization strategy. He initialization ($\text{variance} = 2/n$) is designed for ReLU, accounting for the fact that ReLU zeros out roughly half of its inputs. For GELU or Swish, a slightly different variance is theoretically optimal, but in practice He initialization works well enough for all the smooth activations. Glorot (Xavier) initialization is better suited to sigmoid and tanh and tends to underperform for ReLU-like functions.

Activation functions are what give neural networks their expressive power. Without them, deep networks collapse to linear transforms. Understanding the properties and tradeoffs of each major activation function is fundamental to understanding why modern deep learning architectures are designed the way they are.

The key takeaways from this chapter are:

• Sigmoid and tanh saturate at large inputs, causing vanishing gradients in deep networks. Sigmoid is not zero-centered, which causes zig-zagging during optimization. Tanh is zero-centered with a stronger gradient near zero. Sigmoid is still used for binary output layers; tanh for LSTM and GRU gates.
• ReLU fixed the saturation problem on the positive side, enabling deep networks to be trained for the first time. It induces sparsity and has an identity gradient for positive inputs. Its weakness is dying neurons when neurons consistently receive negative pre-activations.
• Leaky ReLU and PReLU address dying ReLU by allowing a small gradient on the negative side. Leaky ReLU uses a fixed slope (typically 0.01); PReLU learns the optimal slope per neuron.
• ELU and SELU improve on ReLU by restoring a near-zero mean output through an exponential negative side. SELU enables self-normalizing networks under specific initialization and architecture constraints, removing the need for explicit normalization.
• GELU (used in BERT, GPT, and most modern transformers) combines a smooth gate with the normal CDF. This provides gradient flow everywhere and a stochastic regularization interpretation. It is the standard for NLP architectures.
• Swish/SiLU is similar to GELU but uses sigmoid as the gate. It is the activation in EfficientNet and many vision models, and is used in LLaMA-based transformer variants with the gated SwiGLU structure.
• Mish is a smooth, non-monotone activation with competitive performance on vision tasks, particularly object detection architectures.
• Gradient flow through an activation is the most important property for deep network training. The dying neuron problem (zero gradient for negative inputs in ReLU) and vanishing gradients (small maximum gradient in sigmoid and tanh) are the two primary failure modes that modern activations were designed to address.

In the next chapter on Multilayer Perceptrons, we will see how these activation functions fit into the full feedforward architecture: how hidden layers are stacked, how width and depth affect capacity, and how to build and train a complete MLP on a real classification problem.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about activation functions in neural networks.