FFN Activation Functions › Why Nonlinearity Is Non-Negotiable

Before examining specific activation functions, it is worth building a deeper intuition for why nonlinearity matters so much. The feed-forward sublayer in a transformer takes a vector of dimension $d_{\text{model}}$, projects it up to a higher-dimensional space of size $d_{\text{ff}}$, applies an activation, and then projects back down. In mathematical terms, a two-layer FFN computes:

where:

• $\mathbf{x} \in \mathbb{R}^{d_{\text{model}}}$: the input vector from the attention sublayer
• $W_1 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$: the first projection (expands dimensionality)
• $b_1 \in \mathbb{R}^{d_{\text{ff}}}$: the first bias vector
• $\text{act}(\cdot)$: the activation function applied elementwise
• $W_2 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$: the second projection (reduces dimensionality back)
• $b_2 \in \mathbb{R}^{d_{\text{model}}}$: the second bias vector

If $\text{act}$ were the identity function (no nonlinearity), the composition $W_2 \cdot (W_1 \mathbf{x} + b_1) + b_2$ would simplify to $W_2 W_1 \mathbf{x} + W_2 b_1 + b_2$, which is just a linear transformation. Stacking twenty such layers would still produce one linear transformation. All that computation would buy you nothing beyond a single matrix multiply.

The universal approximation theorem tells us that a network with a single hidden layer and a nonlinear activation can approximate any continuous function to arbitrary precision, given enough hidden units. The key requirement is exactly that nonlinearity. In practice, deeper networks with smaller hidden layers learn more efficiently than shallow networks with enormous hidden layers, but the theoretical foundation is the same: nonlinearity is the ingredient that maps matrix multiplication into universal function approximation.

For language modeling specifically, the FFN layers are thought to store factual knowledge. Research by Geva et al. (2021) showed that individual FFN layers function like key-value memories: the first layer's columns act as "keys" that match patterns in the input, and the second layer's rows act as "values" that contribute to the output distribution. The activation function is what allows these keys to be selective, activating strongly for certain input patterns and weakly for others. Without nonlinearity, every key would always partially match every input, making the retrieval process incoherent.

Notice that the FFN's nonlinearity is fundamentally different from attention's nonlinearity. Attention creates nonlinear interactions between positions (which tokens attend to which). The FFN creates nonlinear transformations within each position independently. Together, these two nonlinearities give the transformer its expressive power.