Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Feed-Forward Networks › The Hidden Dimension Expansion

Michael BrenndoerferUpdated June 10, 20257 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 107Feed-Forward Networks › The Hidden Dimension Expansion

Reading section 3 of 13 · The Hidden Dimension Expansion · 7 minSection 3 of 13 · 7 min

Section 3 of 13 · 7 min read

Feed-Forward Networks › The Hidden Dimension ExpansionOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The most striking aspect of the FFN architecture is the dimension expansion. The first linear layer projects from $d_{\text{model}}$ to $d_{ff}$ , typically with $d_{ff} = 4 \times d_{\text{model}}$ . For a model with $d_{\text{model}} = 512$ , this means expanding to 2048 dimensions before projecting back down. For GPT-3 with $d_{\text{model}} = 12288$ , the hidden dimension reaches 49152, nearly fifty thousand neurons per FFN layer.

This expansion might seem wasteful. Why create a 2048-dimensional intermediate representation just to immediately compress it back to 512 dimensions? The answer lies in the expressiveness of the network, and it connects to some deep results in neural network theory.

A fundamental result in approximation theory shows that wider hidden layers can approximate more complex functions. Consider what happens without expansion: if $d_{ff} = d_{\text{model}}$ , the hidden layer has the same dimensionality as the input. While the formula still applies, the network has limited capacity to decompose and recombine features. Each hidden neuron is a linear combination of the input features, gated by nonlinearity. With only $d_{\text{model}}$ neurons, the network can detect $d_{\text{model}}$ different patterns. With $4 \times d_{\text{model}}$ neurons, it can detect four times as many patterns and combine them in far richer ways.

Think of the expansion as temporarily working in a higher-dimensional space where the data is easier to manipulate. In the expanded 2048-dimensional space, the activation function can create very specific, highly discriminating patterns. One hidden neuron might activate only when the input represents a verb in past tense. Another might activate for proper nouns that appear after articles. These fine-grained detectors would be impossible to implement in 512 dimensions because there simply isn't enough room to disentangle all the features. The expansion provides the working space to learn them, and the contraction summarizes the results back into the original dimensionality for the residual connection.

The 4x expansion factor was established in the original "Attention Is All You Need" paper and has become a standard choice. It represents a pragmatic balance between expressiveness and efficiency: large enough to provide substantial nonlinear capacity, small enough to remain computationally tractable. Modern models like LLaMA use slightly different ratios (around 2.7x) when using gated linear units (GLUs), which effectively increase the hidden dimension through gating, but the underlying logic of expanding before contracting remains the same.

In[5]:

Code

# Visualize the dimension flow through the FFN
def analyze_ffn_dimensions(d_model, expansion_factor):
    """Analyze dimensions through FFN layers."""
    d_ff = d_model * expansion_factor

    return {
        "input": d_model,
        "after_W1": d_ff,
        "expansion_ratio": d_ff / d_model,
        "after_W2": d_model,
    }


# Common configurations
configs = [
    ("GPT-2 Small", 768, 4),
    ("GPT-2 Medium", 1024, 4),
    ("GPT-2 Large", 1280, 4),
    ("GPT-3 (175B)", 12288, 4),
    ("LLaMA-7B", 4096, 2.6875),  # Uses 11008 hidden dim
]

Out[6]:

Console

FFN dimension expansion across models:

Model               d_model     d_ff    Ratio
----------------------------------------------
GPT-2 Small             768     3072     4.00x
GPT-2 Medium           1024     4096     4.00x
GPT-2 Large            1280     5120     4.00x
GPT-3 (175B)          12288    49152     4.00x
LLaMA-7B               4096    11008     2.69x

The pattern is consistent: every model expands significantly in the hidden layer. LLaMA's slightly lower ratio reflects the use of the SwiGLU activation, a gated variant that includes an additional linear projection. Even with a nominally lower ratio, SwiGLU-based FFNs have similar or greater effective capacity because the gate mechanism provides additional expressive power.

Let's visualize how the actual activation values change through each stage of the FFN, to build intuition about what the expand-transform-contract process looks like in practice:

Out[7]:

Visualization

Histogram of input values showing normal distribution centered at zero. — Input values follow a normal distribution centered at zero. This reflects the typical distribution of token representations entering the FFN after layer normalization.

Histogram of hidden activations after ReLU showing only positive values and many zeros. — After ReLU, the distribution becomes one-sided: negative values collapse to exactly zero (red line) while positive values pass through unchanged, creating sparsity in the hidden layer.

Histogram of output values showing distribution around zero. — Output values return to a distribution around zero after the second linear projection contracts the sparse hidden representation back to d_model dimensions.

The three histograms tell an important story. The input distribution is centered at zero and roughly Gaussian, which is typical after layer normalization. After ReLU, the distribution becomes one-sided: roughly half the hidden units collapse to exactly zero (shown by the red line), while the other half pass through positive values unchanged. This sparsity is not a bug but a feature: different inputs will activate different subsets of hidden dimensions, creating a sparse code that allows the network to implement many different transformations. The output distribution returns to a balanced form around zero after the second linear layer combines the sparse hidden activations.

Now let's visualize the dimension sizes at each stage with a schematic:

Out[8]:

Visualization

Diagram showing dimension sizes at each FFN layer, with expansion from 512 to 2048 then contraction back to 512. — Dimension flow through the FFN. The input at d_model (512) expands through the first weight matrix W1 to the hidden layer at d_ff (2048, shown as a taller rectangle), passes through the nonlinear activation, then contracts back to d_model (512) via W2. The dramatic size difference illustrates why the 4x expansion dominates the FFN's parameter count.