FFN Activation Functions › The Negative Dip: Feature or Bug?

GELU and SiLU's negative regions deserve closer examination. For certain negative inputs, these activations produce negative outputs rather than zero. This seems counterintuitive: why would we want activations to sometimes invert their input's sign?

The negative dip is a form of implicit regularization. Consider what happens during training: inputs that land in the dip region (roughly $-2 < x < 0$ for GELU) get attenuated but not eliminated. This "soft gating" provides a richer gradient signal than ReLU's hard zero, potentially helping the network escape local minima.

Think of it this way: a neuron in a deep network exists among neighboring neurons, all contributing to the next layer's computation. If one neuron's activation is slightly negative, zeroing it (as ReLU does) eliminates any information that neuron might carry. Allowing it to pass a small negative value (as GELU and SiLU do) preserves that signal, even if attenuated. The downstream layer can then decide whether to amplify or further attenuate it.

The key insight is that the sign of a pre-activation carries information. A pre-activation of $-0.3$ is different from a pre-activation of $-3.0$ in a meaningful way: the first suggests the input is borderline, while the second suggests a strong mismatch with the learned pattern. GELU and SiLU preserve this distinction by scaling the output proportionally. ReLU treats both identically: output zero.

The magnitude of the dip matters for training dynamics. A deeper dip (SiLU) means stronger negative signals for moderately negative inputs, which could either help or hurt depending on the task. Empirically, both work well, with the choice often coming down to which pairs better with other architectural decisions (like gated linear units).

There is also a connection between the negative dip and the activation's non-monotonicity. A monotonic function never decreases as its input increases: ReLU and the standard sigmoid are monotonic. GELU and SiLU are non-monotonic: they decrease slightly in the range around $x \approx -0.5$ before increasing again. This non-monotonicity is unusual for activation functions and was initially viewed with some skepticism. The practical evidence, however, is that non-monotonic activations train comparably to or better than monotonic alternatives at large scale, suggesting that the strict monotonicity constraint imposed by ReLU and its variants was unnecessarily restrictive.