Activation Functions › Summary

Activation functions are what give neural networks their expressive power. Without them, deep networks collapse to linear transforms. Understanding the properties and tradeoffs of each major activation function is fundamental to understanding why modern deep learning architectures are designed the way they are.

The key takeaways from this chapter are:

• Sigmoid and tanh saturate at large inputs, causing vanishing gradients in deep networks. Sigmoid is not zero-centered, which causes zig-zagging during optimization. Tanh is zero-centered with a stronger gradient near zero. Sigmoid is still used for binary output layers; tanh for LSTM and GRU gates.
• ReLU fixed the saturation problem on the positive side, enabling deep networks to be trained for the first time. It induces sparsity and has an identity gradient for positive inputs. Its weakness is dying neurons when neurons consistently receive negative pre-activations.
• Leaky ReLU and PReLU address dying ReLU by allowing a small gradient on the negative side. Leaky ReLU uses a fixed slope (typically 0.01); PReLU learns the optimal slope per neuron.
• ELU and SELU improve on ReLU by restoring a near-zero mean output through an exponential negative side. SELU enables self-normalizing networks under specific initialization and architecture constraints, removing the need for explicit normalization.
• GELU (used in BERT, GPT, and most modern transformers) combines a smooth gate with the normal CDF. This provides gradient flow everywhere and a stochastic regularization interpretation. It is the standard for NLP architectures.
• Swish/SiLU is similar to GELU but uses sigmoid as the gate. It is the activation in EfficientNet and many vision models, and is used in LLaMA-based transformer variants with the gated SwiGLU structure.
• Mish is a smooth, non-monotone activation with competitive performance on vision tasks, particularly object detection architectures.
• Gradient flow through an activation is the most important property for deep network training. The dying neuron problem (zero gradient for negative inputs in ReLU) and vanishing gradients (small maximum gradient in sigmoid and tanh) are the two primary failure modes that modern activations were designed to address.

In the next chapter on Multilayer Perceptrons, we will see how these activation functions fit into the full feedforward architecture: how hidden layers are stacked, how width and depth affect capacity, and how to build and train a complete MLP on a real classification problem.