Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Encoder Architecture › BERT-Style Encoder Configuration

Michael BrenndoerferUpdated June 15, 20254 min read

Language AI HandbookAll chapters

Part 16 of 67Transformer Architectures

Chapter 111Encoder Architecture › BERT-Style Encoder Configuration

Reading section 8 of 13 · BERT-Style Encoder Configuration · 4 minSection 8 of 13 · 4 min

Section 8 of 13 · 4 min read

Encoder Architecture › BERT-Style Encoder ConfigurationOne section · Sections in this chapter8 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

BERT established conventions that influenced nearly all subsequent encoder models. Let's examine the specific architectural choices that defined BERT and its variants.

In[34]:

Code

# BERT configurations
bert_configs = {
    "BERT-Base": {
        "n_layers": 12,
        "d_model": 768,
        "n_heads": 12,
        "d_ff": 3072,  # 4 * d_model
        "vocab_size": 30522,
        "max_position": 512,
    },
    "BERT-Large": {
        "n_layers": 24,
        "d_model": 1024,
        "n_heads": 16,
        "d_ff": 4096,
        "vocab_size": 30522,
        "max_position": 512,
    },
    "RoBERTa-Base": {
        "n_layers": 12,
        "d_model": 768,
        "n_heads": 12,
        "d_ff": 3072,
        "vocab_size": 50265,
        "max_position": 512,
    },
    "ALBERT-xxlarge": {
        "n_layers": 12,
        "d_model": 4096,
        "n_heads": 64,
        "d_ff": 16384,
        "vocab_size": 30000,
        "max_position": 512,
        "embedding_dim": 128,  # ALBERT uses factorized embeddings
        "weight_sharing": True,  # ALBERT shares weights across all layers
    },
}


def count_encoder_params(config):
    """Count parameters in an encoder model."""
    d = config["d_model"]
    ff = config["d_ff"]
    L = config["n_layers"]
    V = config["vocab_size"]
    P = config["max_position"]
    E = config.get("embedding_dim", d)  # ALBERT-style factorization
    weight_sharing = config.get("weight_sharing", False)

    # Embedding parameters
    token_embed = V * E
    position_embed = P * d

    # If using factorized embeddings (ALBERT)
    if E != d:
        embed_projection = E * d
    else:
        embed_projection = 0

    # Per-layer parameters (approximate)
    attention_params = 4 * d * d  # Q, K, V, O projections
    ffn_params = d * ff + ff + ff * d + d  # Two linear layers with biases
    norm_params = 2 * d + 2 * d  # Two layer norms per layer
    layer_params = attention_params + ffn_params + norm_params

    # ALBERT shares one set of weights across all layers (counted once, not L times)
    effective_layers = 1 if weight_sharing else L

    # Total
    total = (
        token_embed
        + position_embed
        + embed_projection
        + effective_layers * layer_params
    )
    return total


# Calculate parameters for each configuration
param_counts = {
    name: count_encoder_params(config) for name, config in bert_configs.items()
}

Out[35]:

Console

Encoder Model Configurations
======================================================================

BERT-Base
----------------------------------------
  Layers: 12
  Hidden size: 768
  Attention heads: 12
  Head dimension: 64
  FFN dimension: 3072 (4x hidden)
  Vocabulary: 30,522
  Max position: 512
  Approx. parameters: 108.9M

BERT-Large
----------------------------------------
  Layers: 24
  Hidden size: 1024
  Attention heads: 16
  Head dimension: 64
  FFN dimension: 4096 (4x hidden)
  Vocabulary: 30,522
  Max position: 512
  Approx. parameters: 334.0M

RoBERTa-Base
----------------------------------------
  Layers: 12
  Hidden size: 768
  Attention heads: 12
  Head dimension: 64
  FFN dimension: 3072 (4x hidden)
  Vocabulary: 50,265
  Max position: 512
  Approx. parameters: 124.0M

ALBERT-xxlarge
----------------------------------------
  Layers: 12
  Hidden size: 4096
  Attention heads: 64
  Head dimension: 64
  FFN dimension: 16384 (4x hidden)
  Vocabulary: 30,000
  Max position: 512
  Embedding dim: 128 (factorized)
  Approx. parameters: 207.8M

Out[36]:

Visualization

Bar chart comparing parameter counts across BERT variants from 100M to 300M parameters. — Parameter counts for common encoder models. BERT-Large roughly quadruples BERT-Base through deeper layers and wider hidden dimensions. ALBERT-xxlarge achieves high capacity with fewer parameters through weight sharing and factorized embeddings.

Key Design PatternsLink Copied

Several architectural choices recur across encoder models.

Head dimension: BERT maintains 64 dimensions per attention head, computed as:

d_k = \frac{d_{\text{model}}}{n_{\text{heads}}}

where:

$d_k$ : the dimension of each attention head's query and key vectors
$d_{\text{model}}$ : the model's hidden dimension (768 for BERT-Base)
$n_{\text{heads}}$ : the number of parallel attention heads (12 for BERT-Base)

This gives $d_k = 768 / 12 = 64$ for BERT-Base. The choice balances expressiveness per head with having multiple heads for diverse attention patterns.

FFN expansion: The feed-forward network uses a 4x expansion factor. With $d_{\text{model}} = 768$ , the FFN hidden dimension is $768 \times 4 = 3072$ . This ratio has become a standard convention. This provides a computational "bottleneck" where most of the model's parameters reside.

Vocabulary and positions: BERT uses WordPiece tokenization with a 30K vocabulary and supports sequences up to 512 tokens. Later models like RoBERTa expand the vocabulary and some variants extend to longer sequences.