Activation Functions › Leaky ReLU and PReLU

Both Leaky ReLU and Parametric ReLU (PReLU) address the dying neuron problem by ensuring that negative pre-activations still produce a nonzero gradient during backpropagation. They differ in whether the negative-side slope is fixed or learned.

Leaky ReLU introduces a small slope on the negative side, so the gradient never completely vanishes:

$$\text{LeakyReLU}(z) = \begin{cases} z & \text{if } z > 0 \\ \alpha z & \text{if } z \leq 0 \end{cases}$$

where:

• $z$: the pre-activation value
• $\alpha$: a small positive constant controlling the negative slope, typically $\alpha = 0.01$

The derivative is:

$$\frac{d\,\text{LeakyReLU}(z)}{dz} = \begin{cases} 1 & \text{if } z > 0 \\ \alpha & \text{if } z \leq 0 \end{cases}$$

With $\alpha = 0.01$, the negative slope sends a gradient of 0.01 back through a dead neuron on every backward pass. The neuron can now gradually revive as its weights receive small but nonzero updates. In practice, Leaky ReLU often trains slightly faster than ReLU and effectively eliminates permanently dead neurons. The tradeoff is a small loss of the sparsity property: neurons no longer output exactly zero for negative inputs, so the regularizing effect of sparsity is reduced.

The choice of $\alpha = 0.01$ is a heuristic. Values between 0.01 and 0.1 are commonly used. Larger $\alpha$ values improve gradient flow more aggressively but at the cost of less sparsity and a less clean identity-like behavior on the positive side. Values above 0.3 start to resemble a linear function more than a rectifier.

Parametric ReLU (PReLU) takes this further by making $\alpha$ a learnable parameter:

$$\text{PReLU}(z) = \begin{cases} z & \text{if } z > 0 \\ \alpha z & \text{if } z \leq 0 \end{cases}$$

where $\alpha$ is now a learned parameter updated by gradient descent alongside the other network weights. Each neuron can have its own $\alpha$, or a single $\alpha$ can be shared across all neurons in a layer.

The gradient with respect to $\alpha$ for a neuron with negative input $z < 0$ is:

$$\frac{\partial L}{\partial \alpha} = \frac{\partial L}{\partial a} \cdot z$$

where:

• $L$: the loss function
• $a$: the neuron's output
• $z$: the pre-activation value (the input to the activation)

PReLU lets the network learn the optimal negative slope for each context. If the data does not benefit from negative activations, the learned $\alpha$ will approach zero, recovering ReLU behavior. If negative activations help, the network will use them. PReLU was introduced in the context of deep residual networks and showed small but consistent improvements over fixed Leaky ReLU in image recognition tasks. The extra parameters introduced by per-neuron $\alpha$ values are typically negligible compared to the weight matrices, so the parameter overhead is minimal.

A related variant, Randomized Leaky ReLU (RReLU), samples $\alpha$ uniformly from a fixed interval during training, then uses a fixed value (the interval's mean) at test time. The stochastic negative slope acts as a regularizer. This provides some of the same benefit as dropout without zeroing entire neurons. RReLU has shown small gains in image recognition benchmarks and is a useful technique to know, though it is less commonly used than Leaky ReLU or PReLU in production architectures.

The following plots show how ReLU, Leaky ReLU, and ELU compare in both their function values and gradients: