Feed-Forward Networks › Summary

The feed-forward network is the workhorse of transformer computation, applying identical transformations to each position independently. This chapter covered its architecture and efficiency, then examined its interpretation within the broader transformer block.

Key takeaways:

• Two-layer architecture: The FFN formula $\text{FFN}(x) = \sigma(xW_1 + b_1)W_2 + b_2$ consists of an expansion layer ($d_{\text{model}} \to d_{ff}$), a nonlinear activation $\sigma$ (ReLU, GELU, etc.), and a contraction layer ($d_{ff} \to d_{\text{model}}$). The standard expansion factor is $d_{ff} = 4 \times d_{\text{model}}$, establishing a working space four times larger than the model dimension where nonlinear transformations are easier to learn.

• Position independence: Each position is processed separately with shared weights. This enables parallel computation and cleanly separates the FFN's role (per-position nonlinear transformation) from attention's role (inter-position information routing). The position independence is not a limitation but a design choice: by the time the FFN processes each token, attention has already incorporated contextual information into that token's representation.

• Key-value memory interpretation: The FFN can be viewed as an associative memory where $W_1$ columns are keys, hidden activations are match scores, and $W_2$ rows are values. Different inputs activate different subsets of hidden dimensions, creating sparse, input-dependent retrievals. This explains how transformers store factual knowledge and why larger FFN dimensions increase a model's capacity to "know" things.

• Dominant parameter count: The FFN contains approximately two-thirds of a transformer block's parameters. With 4x expansion, the total is approximately $8 \times d_{\text{model}}^2$ parameters per layer (from the formula $2 \times d_{\text{model}} \times d_{ff} + d_{ff} + d_{\text{model}}$). This quadratic scaling with $d_{\text{model}}$ makes FFN optimization critical for model efficiency.

• Linear computational scaling: FFN compute cost is $4 \times n \times d_{\text{model}} \times d_{ff}$ FLOPs, scaling linearly with sequence length $n$. This contrasts with attention's quadratic $O(n^2)$ scaling. For short sequences, FFN dominates compute; for long sequences, attention becomes the bottleneck.

• Crossover point: Around 1000-2000 tokens, FFN and attention have comparable computational costs for typical model sizes. This crossover point shapes optimization strategy: short-context systems prioritize FFN compression, while long-context systems prioritize attention efficiency.

The next chapter examines activation functions used in FFNs in depth, comparing ReLU, GELU, SiLU/Swish, and their gated variants (GLU, SwiGLU), and exploring why modern models have moved beyond the original ReLU choice toward smoother, more expressive alternatives.