Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BERT Architecture › Model Sizes: Base vs Large

Michael BrenndoerferUpdated July 16, 20256 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 169BERT Architecture › Model Sizes: Base vs Large

Reading section 2 of 12 · Model Sizes: Base vs Large · 6 minSection 2 of 12 · 6 min

Section 2 of 12 · 6 min read

BERT Architecture › Model Sizes: Base vs LargeOne section · Sections in this chapter2 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

BERT comes in two standard configurations: BERT-Base and BERT-Large. These sizes were chosen deliberately to balance capability against practical deployment constraints. BERT-Base matches the hidden dimension of OpenAI's GPT (768), enabling direct comparisons, while BERT-Large tests whether larger models capture finer language patterns.

The choice to release two sizes reflects the practical reality that research and production have different requirements. BERT-Base was designed to be large enough to achieve strong performance while still being trainable and deployable on hardware available to most researchers and companies in 2018. BERT-Large was designed as a proof of concept: does more scale help? The answer was unambiguously yes. BERT-Large consistently outperformed BERT-Base by several points on most benchmarks, establishing a pattern that would define the field for years: more parameters and more layers generally produce better representations.

Notice that both models use a head dimension of exactly 64. This is not a coincidence. The head dimension determines the size of the QKV vectors within each attention head. At 64 dimensions, these vectors are large enough to express complex patterns but small enough that the dot products between query and key vectors remain numerically stable without aggressive scaling. The Vaswani et al. transformer paper argued that head dimensions should be in this range for stable training, and BERT adopted this guidance. When BERT-Large increases from 12 to 16 heads, it achieves greater representational capacity by running more parallel attention computations rather than by making each computation larger.

BERT Model Variants

BERT-Base contains 12 transformer encoder layers with 12 attention heads and a hidden dimension of 768, totaling approximately 110 million parameters. BERT-Large doubles the layers to 24, increases heads to 16, and expands the hidden dimension to 1024, reaching approximately 340 million parameters.

The architectural specifications for each variant are:

BERT architectural specifications. The head dimension remains constant at 64 across both variants, with BERT-Large achieving greater capacity through more heads and layers.
Parameter	BERT-Base	BERT-Large
Layers ( $L$ )	12	24
Hidden size ( $H$ )	768	1024
Attention heads ( $A$ )	12	16
Head dimension ( $H/A$ )	64	64
Feed-forward size	3072	4096
Vocabulary size	30,522	30,522
Max sequence length	512	512
Parameters	~110M	~340M

Notice that the head dimension remains constant at 64 across both variants. This means BERT-Large achieves more capacity by having more heads (16 vs 12) and more layers (24 vs 12), not by making each head larger. The feed-forward dimension follows the standard 4x multiplier relative to hidden size (768 × 4 = 3072 for Base, 1024 × 4 = 4096 for Large).

Let's compute the exact parameter counts to understand where capacity resides:

In[4]:

Code

def count_bert_parameters(
    vocab_size: int = 30522,
    hidden_size: int = 768,
    num_layers: int = 12,
    num_heads: int = 12,
    intermediate_size: int = 3072,
    max_position: int = 512,
    type_vocab_size: int = 2,
) -> dict:
    """Count parameters in each component of BERT."""
    params = {}

    # Embedding layers
    params["token_embeddings"] = vocab_size * hidden_size
    params["position_embeddings"] = max_position * hidden_size
    params["segment_embeddings"] = type_vocab_size * hidden_size
    params["embedding_layernorm"] = 2 * hidden_size  # gamma and beta

    # Per-layer parameters
    # Self-attention: Q, K, V projections + output projection
    attention_params = 4 * (hidden_size * hidden_size + hidden_size)
    # Feed-forward: two linear layers
    ff_params = 2 * (hidden_size * intermediate_size + intermediate_size)
    ff_params += hidden_size  # output bias
    # Layer norms (2 per layer)
    layernorm_params = 4 * hidden_size

    params["per_layer"] = attention_params + ff_params + layernorm_params
    params["all_layers"] = num_layers * params["per_layer"]

    # Pooler (for [CLS] representation)
    params["pooler"] = hidden_size * hidden_size + hidden_size

    # Total
    params["embeddings_total"] = (
        params["token_embeddings"]
        + params["position_embeddings"]
        + params["segment_embeddings"]
        + params["embedding_layernorm"]
    )
    params["total"] = (
        params["embeddings_total"] + params["all_layers"] + params["pooler"]
    )

    return params

Out[5]:

Console

BERT-Base Parameter Distribution:
  Embeddings: 23,837,184 (21.8%)
  Transformer Layers: 85,091,328 (77.7%)
  Pooler: 590,592 (0.5%)
  Total: 109,519,104

BERT-Large Parameter Distribution:
  Embeddings: 31,782,912 (9.5%)
  Transformer Layers: 302,407,680 (90.2%)
  Pooler: 1,049,600 (0.3%)
  Total: 335,240,192

Out[6]:

Visualization

Stacked bar chart comparing parameter distribution between BERT-Base and BERT-Large. — Parameter distribution across BERT components. Transformer layers dominate in both variants, but their share grows from 78% to 90% as model size increases, while embeddings become proportionally smaller.

The vast majority of parameters reside in the transformer layers, particularly in the feed-forward networks. Token embeddings represent a significant portion due to the large vocabulary, but their contribution decreases proportionally as model depth increases. This distribution matters for understanding where BERT stores knowledge: factual information tends to concentrate in feed-forward weights, while attention patterns encode syntactic and semantic relationships.

Historical Context: Why These Numbers?

The BERT-Base configuration was not chosen arbitrarily. The 12-layer, 12-head, 768-hidden-size specification matches OpenAI GPT's architecture almost exactly. The Google BERT team made this choice deliberately to enable an apples-to-apples comparison: with identical architecture but different training objectives (bidirectional MLM plus NSP versus left-to-right LM), any performance difference can be attributed to the training approach rather than model capacity. BERT-Base's outperformance of GPT on most benchmarks showed that bidirectional pretraining was more useful for these understanding tasks, even with identical model size.

In practice, the choice between BERT-Base and BERT-Large often comes down to deployment constraints rather than performance requirements. BERT-Large requires roughly 3x more memory and compute than BERT-Base. For applications where latency and cost are important, BERT-Base frequently offers 90-95% of BERT-Large's performance at one-third the cost. This tradeoff has motivated research into distilled versions like DistilBERT, which achieves about 97% of BERT-Base's performance at 40% of its size. The next chapter covers these variants in detail.