Activation Functions › Activation Functions

In the previous chapter on Linear Classifiers, we saw how a network layer computes a weighted sum of its inputs and passes the result through a softmax function to produce class probabilities. That operation is inherently linear: no matter how many such layers you stack, the composition of linear functions is still a linear function. A 100-layer network without anything nonlinear between the layers is mathematically identical to a single matrix multiplication. Activation functions break this constraint. They are the nonlinear gates placed after each linear transformation, and without them deep networks would have no more expressive power than a single-layer perceptron.

To appreciate why this matters, consider what a linear layer does. A linear transformation maps the input space to an output space through rotation, scaling, and reflection. You can stack as many of these transformations as you like, but the final result is still just another rotation, scaling, and reflection of the original input. There is no way to separate points that require a curved boundary, no way to model interactions between features that are not already captured by a single weighted sum. The moment you insert a nonlinear activation function between two linear layers, everything changes. Now the network can represent curved decision boundaries, and as you add more layers with more nonlinear activations, the representable function class grows exponentially.

Choosing the right activation function is not a cosmetic decision. The function you pick determines how gradients flow backward through the network during training, how quickly weights converge, whether certain neurons become permanently inactive, and how the network behaves in deep architectures. This chapter covers every major activation function used in modern neural networks: from the classical sigmoid and tanh through ReLU and its variants, through the exponential families ELU and SELU, all the way to the smooth modern functions like GELU, Swish, and Mish that power today's language models.