Activation Functions › ReLU and the Dying ReLU Problem

The Rectified Linear Unit (ReLU) was proposed as a simple solution to saturation. By making the function piecewise linear, it guarantees that the gradient on the positive side is always exactly 1, no matter how large the input. This is the key reason deep networks became trainable in practice.

ReLU is defined as:

$$\text{ReLU}(z) = \max(0, z) = \begin{cases} z & \text{if } z > 0 \\ 0 & \text{if } z \leq 0 \end{cases}$$

where:

• $z$: the pre-activation value
• $\max(0, z)$: takes the larger of 0 and $z$, clipping negative values to zero

The derivative is straightforward:

$$\frac{d\,\text{ReLU}(z)}{dz} = \begin{cases} 1 & \text{if } z > 0 \\ 0 & \text{if } z < 0 \end{cases}$$

At $z = 0$, the derivative is technically undefined, but in practice implementations return 0 or 1 here without issue.

The gradient being 1 for all positive inputs eliminates saturation on the positive side entirely. When a neuron is active (positive input), gradients flow through it unchanged. This makes training deep networks with ReLU dramatically faster than with sigmoid or tanh. ReLU was a key enabler of the deep learning revolution in the early 2010s, particularly for convolutional networks like AlexNet (2012) and VGG (2014), and it remains the default activation for most hidden layers in feedforward and convolutional networks today.

ReLU also has a property called sparsity induction: when many neurons have negative pre-activation inputs, they output exactly zero. For a typical network with random weights early in training, roughly half the neurons are inactive on any given input. This sparsity means fewer neurons participate in each forward pass, which has two practical benefits. First, it reduces co-adaptation between neurons: neurons cannot rely on the output of another neuron that is often inactive, which acts as an implicit regularizer. Second, sparse activations are computationally efficient, particularly for hardware accelerators that can skip zero-valued computations.

There is also a biological motivation for ReLU-like functions. Neurons in the brain have a firing threshold: below a certain stimulus level, they produce no response. Above it, they fire at a rate proportional to the stimulus. This is qualitatively similar to ReLU, though the analogy should not be taken too literally.

ReLU's left side is its weakness. Any neuron that receives a negative pre-activation outputs zero and contributes nothing to the forward pass. More critically, its gradient is also zero, so no update flows back through it. If a neuron consistently receives negative inputs across the entire training set, its weights stop updating entirely. The neuron is dead.

This happens in practice more often than expected. A large learning rate can push the weights in one training step such that a neuron's pre-activation becomes negative for all training examples. Once that happens, the neuron's gradient is zero for all examples, and its weights receive no signal to move back. The dead state is an absorbing state: once a neuron dies, it cannot recover on its own.

Networks trained with ReLU can have 10-40% of neurons permanently dead, depending on architecture, learning rate, and weight initialization. With a learning rate that is too high, a large weight update can drive many neurons negative simultaneously. With poor initialization (weights that are too large), the same effect can happen at the start of training. The dying ReLU problem is most severe in early layers because those neurons receive less direct signal from the loss.

Several practical measures reduce the risk:

• Use a small learning rate, especially early in training
• Initialize weights with He initialization ($\text{variance} = 2/n$, where $n$ is the fan-in), which keeps pre-activations well-distributed around zero
• Use gradient clipping to prevent large weight updates
• Monitor the fraction of dead neurons during training as a diagnostic

The fix at the architectural level is to ensure that at least some gradient can flow even when $z < 0$. This led to several ReLU variants.