Activation Functions › ELU and SELU

Exponential Linear Units address a different problem from the dying neuron issue. Even when ReLU neurons are alive, the average activation of a ReLU layer is always positive (since ReLU clips negatives to zero). This causes the same bias-shift problem that sigmoid suffers from: if all outputs from a layer are positive, the gradients for all weights in the next layer must have the same sign, leading to zigzag optimization paths. ELU restores a mean closer to zero by giving the negative side a smooth exponential shape rather than a hard zero.

ELU is defined as:

$$\text{ELU}(z) = \begin{cases} z & \text{if } z > 0 \\ \alpha(e^z - 1) & \text{if } z \leq 0 \end{cases}$$

where:

• $z$: the pre-activation value
• $\alpha$: a positive hyperparameter (typically $\alpha = 1$) controlling the saturation point for negative inputs
• $e^z$: exponential of the pre-activation, which smoothly approaches $-\alpha$ as $z \to -\infty$

The derivative is:

$$\frac{d\,\text{ELU}(z)}{dz} = \begin{cases} 1 & \text{if } z > 0 \\ \text{ELU}(z) + \alpha & \text{if } z \leq 0 \end{cases}$$

For large negative inputs, ELU approaches $-\alpha$, so the mean output of an ELU layer can be zero if the distribution of pre-activations is approximately symmetric around zero. This zero-mean property means the gradients for weights feeding into the next layer do not suffer from the systematic sign bias that ReLU creates. ELU is more computationally expensive than ReLU because of the exponential computation, but in architectures where zero-centering matters (particularly deep feedforward networks trained without batch normalization) it often converges faster.

The key difference between ELU and Leaky ReLU is the shape of the negative side. Leaky ReLU uses a linear negative side (a straight line with slope $\alpha$). ELU uses an exponential that starts at 0 when $z = 0$ and saturates at $-\alpha$ as $z \to -\infty$. The saturation means ELU is more less brittle to very negative inputs than Leaky ReLU, which grows without bound in the negative direction. This bounded negative side is important for mean stability.

Scaled Exponential Linear Unit (SELU) takes ELU further with a specific pair of scaling constants $\lambda$ and $\alpha$ chosen to enforce self-normalizing behavior:

$$\text{SELU}(z) = \lambda \begin{cases} z & \text{if } z > 0 \\ \alpha(e^z - 1) & \text{if } z \leq 0 \end{cases}$$

where:

• $\lambda \approx 1.0507$: the outer scaling constant
• $\alpha \approx 1.6733$: the negative saturation constant

These specific values were derived analytically by Klambauer et al. (2017) to ensure that if the inputs to a SELU layer are approximately standardized (mean 0, variance 1), the outputs will also be approximately standardized, regardless of network depth. This is the self-normalizing property: SELU networks tend to maintain stable activations throughout training without explicit batch normalization.

The self-normalization property is remarkable because it removes the need for one of the most important regularization techniques in deep learning. Batch normalization is computationally expensive, requires careful tuning, and interacts poorly with small batch sizes and certain architectures. SELU achieves similar statistical guarantees through the activation function itself.

SELU works best under specific conditions:

• Weights initialized with LeCun normal initialization ($\text{mean} = 0$, $\text{variance} = 1/n$, where $n$ is the number of input units)
• No batch normalization or dropout (which would break the statistical properties SELU relies on)
• Fully connected layers (convolutional layers need extra care because the spatial structure creates dependencies that violate SELU's assumptions)
• Data that is approximately standardized before entering the network

When conditions are met, SELU can train deep fully connected networks reliably without normalization layers. In practice, it is used less often than ReLU because transformers and convolutional architectures use different normalization strategies, but it remains relevant for structured and tabular deep learning where batch normalization interactions are problematic or where small batch sizes make batch statistics unreliable.