Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Architecture Hyperparameters › The Core Hyperparameters

Michael BrenndoerferUpdated June 18, 20255 min read

Language AI HandbookAll chapters

Part 16 of 67Transformer Architectures

Chapter 116Architecture Hyperparameters › The Core Hyperparameters

Reading section 2 of 10 · The Core Hyperparameters · 5 minSection 2 of 10 · 5 min

Section 2 of 10 · 5 min read

Architecture Hyperparameters › The Core HyperparametersOne section · Sections in this chapter2 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Every transformer architecture is defined by a small set of core hyperparameters that determine its capacity and computational requirements. Understanding these parameters and their interactions is essential for both implementing existing architectures and designing new ones.

The Six Core Hyperparameters

A transformer's architecture is primarily determined by six numbers: vocabulary size ( $V$ ), number of layers ( $L$ ), model dimension ( $d_{\text{model}}$ ), number of attention heads ( $n_{\text{heads}}$ ), head dimension ( $d_{\text{head}}$ ), and feed-forward dimension ( $d_{\text{ff}}$ ). Everything else, including parameter count, memory usage, and computational cost, derives from these choices.

Let's define each parameter precisely:

$V$ (vocabulary size): The number of unique tokens the model can process. Typical values range from 30,000 (BERT) to 100,000 (GPT-4 class models). Larger vocabularies reduce sequence length but increase embedding memory.
$L$ (number of layers): How many transformer blocks are stacked. Also called model depth. Values range from 6 (small models) to 96+ (frontier models). More layers provide more representational capacity but increase compute and memory linearly.
$d_{\text{model}}$ (model dimension): The hidden dimension that flows through the entire model. Also called embedding dimension or hidden size. This dimension is preserved throughout the transformer, from embeddings through attention and FFN layers. Common values: 768 (BERT-base), 1024 (BERT-large), 4096 (LLaMA-7B), 8192 (LLaMA-70B).
$n_{\text{heads}}$ (number of attention heads): How many parallel attention operations run in each layer. The model dimension must be evenly divisible by the number of heads. More heads allow the model to attend to different types of relationships simultaneously.
$d_{\text{head}}$ (head dimension): The dimension of QKV representations within each attention head. Computed as $d_{\text{head}} = d_{\text{model}} / n_{\text{heads}}$ in standard multi-head attention. Modern architectures sometimes set this independently, often to 64 or 128.
$d_{\text{ff}}$ (feed-forward dimension): The hidden dimension of the feed-forward network within each block. Traditionally set to $4 \times d_{\text{model}}$ . For SwiGLU-based FFNs, typically $\frac{8}{3} \times d_{\text{model}}$ to maintain similar parameter count.

In[3]:

Code

import numpy as np

np.random.seed(42)

# Example configurations from well-known models
model_configs = {
    "BERT-base": {
        "V": 30522,
        "L": 12,
        "d_model": 768,
        "n_heads": 12,
        "d_ff": 3072,
    },
    "BERT-large": {
        "V": 30522,
        "L": 24,
        "d_model": 1024,
        "n_heads": 16,
        "d_ff": 4096,
    },
    "GPT-2 Small": {
        "V": 50257,
        "L": 12,
        "d_model": 768,
        "n_heads": 12,
        "d_ff": 3072,
    },
    "GPT-2 XL": {
        "V": 50257,
        "L": 48,
        "d_model": 1600,
        "n_heads": 25,
        "d_ff": 6400,
    },
    "LLaMA-7B": {
        "V": 32000,
        "L": 32,
        "d_model": 4096,
        "n_heads": 32,
        "d_ff": 11008,  # SwiGLU: ~2.7x
    },
    "LLaMA-70B": {
        "V": 32000,
        "L": 80,
        "d_model": 8192,
        "n_heads": 64,
        "d_ff": 28672,
    },
}

# Compute derived parameters
for name, config in model_configs.items():
    config["d_head"] = config["d_model"] // config["n_heads"]

Out[4]:

Console

Model configurations with derived head dimensions:

Model            Layers  d_model  Heads  d_head     d_ff d_ff ratio
----------------------------------------------------------------------
BERT-base            12      768     12      64     3072       4.00x
BERT-large           24     1024     16      64     4096       4.00x
GPT-2 Small          12      768     12      64     3072       4.00x
GPT-2 XL             48     1600     25      64     6400       4.00x
LLaMA-7B             32     4096     32     128    11008       2.69x
LLaMA-70B            80     8192     64     128    28672       3.50x

Notice the patterns across these models. Head dimensions cluster around 64 to 128, suggesting this range provides a good balance between per-head expressiveness and enabling multiple heads. The FFN ratio is consistently around 4x for traditional architectures (BERT, GPT-2) but closer to 2.7x for SwiGLU-based models (LLaMA) because the gating mechanism effectively doubles the transformation, requiring less explicit expansion.

Out[5]:

Visualization

Scatter plot showing transformer models positioned by layers and model dimension, with bubble size proportional to parameters. — Major transformer architectures plotted in depth × width space. Bubble size indicates total parameter count. Models generally follow a diagonal pattern, scaling both dimensions together rather than maximizing one at the expense of the other.

This scatter plot reveals that successful architectures don't maximize just one dimension. Instead, they follow a roughly linear relationship between depth and width, scaling both together. The diagonal trend suggests an empirical consensus about optimal proportions.