Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Transformer Block Assembly › Comparing Block Architectures

Michael BrenndoerferUpdated June 13, 20253 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 110Transformer Block Assembly › Comparing Block Architectures

Reading section 10 of 16 · Comparing Block Architectures · 3 minSection 10 of 16 · 3 min

Section 10 of 16 · 3 min read

Transformer Block Assembly › Comparing Block ArchitecturesOne section · Sections in this chapter10 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Different models use slightly different block configurations. Let's compare the major variants:

In[33]:

Code

# Define configurations for major architectures
architectures = {
    "GPT-2": {
        "norm_type": "LayerNorm",
        "norm_placement": "Pre-norm",
        "ffn_type": "Standard (GELU)",
        "ffn_expansion": 4.0,
        "attention_type": "Dense",
    },
    "LLaMA": {
        "norm_type": "RMSNorm",
        "norm_placement": "Pre-norm",
        "ffn_type": "SwiGLU",
        "ffn_expansion": 2.7,  # ~8/3 with gating
        "attention_type": "Dense + RoPE",
    },
    "Mistral": {
        "norm_type": "RMSNorm",
        "norm_placement": "Pre-norm",
        "ffn_type": "SwiGLU",
        "ffn_expansion": 2.7,
        "attention_type": "Sliding Window + GQA",
    },
    "BERT": {
        "norm_type": "LayerNorm",
        "norm_placement": "Post-norm",
        "ffn_type": "Standard (GELU)",
        "ffn_expansion": 4.0,
        "attention_type": "Dense (bidirectional)",
    },
}

Out[34]:

Console

Transformer Block Architecture Comparison
======================================================================

Component           GPT-2       LLaMA       Mistral     BERT        
----------------------------------------------------------------------
Norm Type           LayerNorm   RMSNorm     RMSNorm     LayerNorm   
Norm Placement      Pre-norm    Pre-norm    Pre-norm    Post-norm   
Ffn Type            Standard (GELU)SwiGLU      SwiGLU      Standard (GELU)
Ffn Expansion       4.0         x2.7         x2.7         x4.0         x
Attention Type      Dense       Dense + RoPESliding Window + GQADense (bidirectional)

The table shows main design decisions that distinguish architectures:

Modern models (LLaMA, Mistral) use RMSNorm for efficiency and SwiGLU for improved FFN expressiveness
BERT is a notable exception, using post-norm placement (it predates the pre-norm stability insights)
FFN expansion is smaller with gated variants because gating effectively doubles the hidden dimension

To visualize how these architectural choices affect parameter allocation, let's compare parameter counts across different model scales:

Out[35]:

Visualization

Stacked area chart showing parameter counts for embedding, attention, and FFN components across different d_model values. — How transformer parameters scale with model size. As d_model increases, attention (which scales as d_model squared) and FFN (which scales as d_model times d_ff) grow at different rates. The embedding layer's relative contribution decreases for larger models with fixed vocabulary size.

The scaling chart reveals an important insight: as models grow larger, the FFN becomes increasingly dominant. At $d_{\text{model}} = 4096$ , the FFN accounts for over 80% of transformer layer parameters. This explains why efficiency innovations (like Mixture of Experts) primarily target the FFN component.