Feed-Forward Networks › Feed-Forward Networks
Language AI HandbookAll chapters Feed-Forward Networks › Feed-Forward NetworksOne section · Sections in this chapter1 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Self-attention lets tokens gather contextual information from across the sequence. But attention alone is limited: it computes weighted averages of value vectors, a fundamentally linear operation. After softmax normalization, each output position is a convex combination of value vectors. No matter how many attention heads you stack, no matter how carefully you tune the query and key projections, the mapping from input to output remains linear in the values. To learn complex functions of language, transformers need something more.
The feed-forward network (FFN) provides exactly this missing ingredient. Sitting in every transformer block alongside the self-attention sublayer, it applies a nonlinear transformation to each token's representation independently. Think of it as the transformer's "thinking" layer: after attention has routed information between positions and assembled contextually-aware representations, the FFN processes each of those representations through a small neural network that can learn curved, high-dimensional decision boundaries. This is where the actual computational work of language understanding happens.
Every transformer block contains two main components: a self-attention sublayer and a feed-forward sublayer. While attention handles inter-token communication, the FFN handles per-token computation. The division of labor is elegant and deliberate. Attention routes information: it reads from the full sequence and assembles each position's context. The FFN transforms information: it takes each position's assembled context and applies a learned nonlinear function to produce a richer representation. Together, they enable transformers to learn the rich, hierarchical, compositional representations that power modern language AI.
Understanding the FFN matters beyond academic interest. In most transformer architectures, the FFN contains significantly more parameters than the attention mechanism, often two-thirds or more of each layer's total parameters. It also dominates computational cost for short sequences. This combination, large parameter count and heavy compute, makes FFN design and optimization one of the most important engineering problems in modern LLM deployment. Techniques like mixture of experts, activation sparsity, and quantization all target the FFN specifically.
The position-wise feed-forward network was introduced in the original transformer paper "Attention Is All You Need" by Vaswani et al. (2017). The authors used a two-layer architecture with ReLU activation and a 4x hidden dimension expansion factor, choices that have become de facto standards across the field. What started as a practical design decision, pairing a nonlinear sublayer with the attention sublayer, turned out to encode deep inductive biases about how information should be processed in language models. Subsequent research has refined the activation function (ReLU to GELU to SiLU/Swish), explored gated variants (GLU, SwiGLU), and investigated the FFN's role as associative memory, but the basic two-layer expand-and-contract structure has remained remarkably stable across nearly a decade of architectural innovation.
This chapter examines the FFN in detail. You'll learn its two-layer architecture and understand why the hidden dimension is expanded. You'll see how position independence enables massive parallelization and how the FFN acts as an associative memory that stores factual knowledge in its weights. You'll work through a numerical example step-by-step, trace through a complete implementation, and calculate the substantial parameter count that makes FFNs the largest component of most transformer models. By the end, you'll understand what the FFN computes, why it's designed the way it is, and what would be lost without it.