Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Feed-Forward Networks › Parameter Count Analysis

Michael BrenndoerferUpdated June 10, 20257 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 107Feed-Forward Networks › Parameter Count Analysis

Reading section 6 of 13 · Parameter Count Analysis · 7 minSection 6 of 13 · 7 min

Section 6 of 13 · 7 min read

Feed-Forward Networks › Parameter Count AnalysisOne section · Sections in this chapter6 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Feed-forward networks are the largest component of transformer models by parameter count. In a standard transformer, the FFN contains significantly more parameters than the attention mechanism. This matters for model scaling and efficiency optimization, and it helps explain why so many recent research directions (mixture of experts, weight sharing, factorization) target the FFN specifically.

For a single FFN layer, we count the total number of learnable parameters by summing the sizes of all weight matrices and bias vectors. The calculation is straightforward. Matrix $W_1$ has shape $d_{\text{model}} \times d_{ff}$ , contributing $d_{\text{model}} \times d_{ff}$ parameters. Matrix $W_2$ has shape $d_{ff} \times d_{\text{model}}$ , contributing an equal $d_{ff} \times d_{\text{model}}$ parameters. The two bias vectors contribute $d_{ff}$ and $d_{\text{model}}$ parameters respectively. Adding these together:

\text{FFN parameters} = 2 \times d_{\text{model}} \times d_{ff} + d_{ff} + d_{\text{model}}

where:

$d_{\text{model}} \times d_{ff}$ : the number of elements in weight matrix $W_1$ (rows $\times$ columns)
$d_{ff} \times d_{\text{model}}$ : the number of elements in weight matrix $W_2$ (rows $\times$ columns, equal to the count in $W_1$ )
$d_{ff}$ : the number of elements in bias vector $b_1$
$d_{\text{model}}$ : the number of elements in bias vector $b_2$
The factor of 2 in front of $d_{\text{model}} \times d_{ff}$ accounts for both weight matrices $W_1$ and $W_2$ having the same total element count

With the standard expansion factor $d_{ff} = 4 \times d_{\text{model}}$ , we can simplify this expression. Substituting $d_{ff} = 4 d_{\text{model}}$ :

\begin{aligned} \text{FFN parameters} &= 2 \times d_{\text{model}} \times (4 \times d_{\text{model}}) + (4 \times d_{\text{model}}) + d_{\text{model}} \\ &= 8 \times d_{\text{model}}^2 + 5 \times d_{\text{model}} \end{aligned}

For large models where $d_{\text{model}}$ is in the hundreds or thousands, the quadratic term dominates and the linear bias terms become negligible, giving approximately $8 \times d_{\text{model}}^2$ parameters per FFN layer.

Why does this formula make sense? Notice that the parameter count grows quadratically with $d_{\text{model}}$ : if you double the model dimension, you quadruple the FFN parameter count. This is the same quadratic scaling that makes attention expensive with respect to sequence length, but here the quadratic factor is in the model dimension rather than sequence length. Doubling $d_{\text{model}}$ from 512 to 1024 increases FFN parameters from roughly 2 million to 8 million per layer, a 4x increase for a 2x dimension increase.

In[15]:

Code

def count_ffn_params(d_model, d_ff, include_bias=True):
    """Count parameters in a feed-forward network."""
    weight_params = 2 * d_model * d_ff  # W1 and W2
    bias_params = d_ff + d_model if include_bias else 0
    return weight_params + bias_params


def count_attention_params(d_model, num_heads, include_bias=True):
    """Count parameters in multi-head attention."""
    # Q, K, V projections and output projection
    weight_params = 4 * d_model * d_model  # W_Q, W_K, W_V, W_O
    bias_params = 4 * d_model if include_bias else 0
    return weight_params + bias_params


# Compare for different model sizes
model_configs = [
    ("GPT-2 Small", 768, 3072, 12),
    ("GPT-2 Medium", 1024, 4096, 16),
    ("BERT-Base", 768, 3072, 12),
    ("GPT-3 (175B)", 12288, 49152, 96),
]

Out[16]:

Console

Parameter count comparison: FFN vs Attention (per layer)

Model            d_model     d_ff     FFN Params    Attn Params   FFN/Attn
--------------------------------------------------------------------------------
GPT-2 Small          768     3072      4,718,592      2,359,296        2.0x
GPT-2 Medium        1024     4096      8,388,608      4,194,304        2.0x
BERT-Base            768     3072      4,718,592      2,359,296        2.0x
GPT-3 (175B)       12288    49152  1,207,959,552    603,979,776        2.0x

The FFN consistently contains about twice as many parameters as the attention mechanism per layer. This 2:1 ratio emerges from a straightforward calculation: attention uses four weight matrices of size $d_{\text{model}} \times d_{\text{model}}$ for QKV and output projections, totaling $4 d_{\text{model}}^2$ parameters. The FFN uses two matrices with total size $2 d_{\text{model}} \times d_{ff} = 2 d_{\text{model}} \times 4 d_{\text{model}} = 8 d_{\text{model}}^2$ parameters. The ratio is exactly $8/4 = 2$ , so the FFN always has twice the weight parameters of attention when the 4x expansion factor is used.

For large models like GPT-3, each transformer layer has over 2.4 billion parameters in the FFN alone. Multiplied across 96 layers, the FFN accounts for the vast majority of the model's 175 billion total parameters. This makes FFN optimization critical for model efficiency, both in terms of memory and computation.

Let's visualize how FFN parameters scale with model dimension:

Out[17]:

Visualization

Line plot showing FFN parameters increasing quadratically from millions to billions as d_model increases from 256 to 12288. — FFN parameter count (blue circles) scales quadratically with model dimension, shown here on a log-log scale. The dashed red line shows the approximation 8 times d_model squared, which matches the exact count closely for large d_model values where bias terms are negligible. Doubling d_model quadruples the parameter count, which is why large language models devote such significant engineering effort to FFN compression.

Let's also visualize the parameter distribution in a transformer block to see the FFN's dominance at a glance:

Out[18]:

Visualization

Pie chart showing FFN with ~67% of parameters, attention with ~33%, and layer norm with <1%. — Parameter distribution across components in a single GPT-2 Small transformer block (d_model=768). The feed-forward network accounts for roughly two-thirds of all learnable parameters, with the attention mechanism holding most of the remainder. Layer normalization parameters are negligible in comparison. This distribution motivates the heavy focus on FFN optimization in efficient transformer research.