BERT Pre-training › Pre-training Hyperparameters
Language AI HandbookAll chapters BERT Pre-training › Pre-training HyperparametersOne section · Sections in this chapter6 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT's pre-training configuration was carefully tuned. The hyperparameters balance training stability, convergence speed, and final model quality.
Optimizer Settings
BERT uses AdamW, a variant of Adam that decouples weight decay from the gradient update:
- Learning rate: Peak of 1e-4 for both BERT-base and BERT-large
- Warmup steps: 10,000 steps of linear warmup from 0 to peak learning rate
- Learning rate schedule: Linear decay from peak to 0 over the remaining training
- Weight decay: 0.01 applied to all parameters except biases and LayerNorm
- Adam : 0.9
- Adam : 0.999
- Adam : 1e-6
The warmup phase is critical for training stability. At initialization, gradients can be very large. Starting with a high learning rate would cause the model to diverge. Warmup gradually increases the learning rate, allowing the model to find a reasonable parameter region before taking larger steps.
Batch Size and Sequence Length
BERT uses large batch sizes to improve training efficiency:
- Batch size: 256 sequences
- Sequence length: 128 tokens for 90% of training, 512 tokens for final 10%
- Effective tokens per batch: 256 × 128 = 32,768 tokens (short phase), 256 × 512 = 131,072 tokens (long phase)
The two-phase approach saves compute. Most learning happens with short sequences where attention is cheap. The final phase with long sequences teaches the model to handle extended context.
# Compute training statistics
batch_size = 256
short_seq_len = 128
long_seq_len = 512
total_steps = 1_000_000
short_phase_ratio = 0.9
short_phase_steps = int(total_steps * short_phase_ratio)
long_phase_steps = total_steps - short_phase_steps
short_phase_tokens = short_phase_steps * batch_size * short_seq_len
long_phase_tokens = long_phase_steps * batch_size * long_seq_len
total_tokens = short_phase_tokens + long_phase_tokensBERT Pre-training Statistics: Short sequence phase (seq_len=128): Steps: 900,000 Tokens: 29,491,200,000 (29.5B) Long sequence phase (seq_len=512): Steps: 100,000 Tokens: 13,107,200,000 (13.1B) Total tokens processed: 42,598,400,000 (42.6B)
BERT-base processes approximately 40 billion tokens during pre-training. This is roughly 12 passes through the 3.3 billion word corpus, allowing the model to see each token multiple times with different masking patterns.
Model Configurations
BERT comes in two sizes:
| Parameter | BERT-base | BERT-large |
|---|---|---|
| Layers | 12 | 24 |
| Hidden size | 768 | 1024 |
| Attention heads | 12 | 16 |
| Feed-forward size | 3072 | 4096 |
| Parameters | 110M | 340M |
| Pre-training time | ~4 days | ~12 days |
The larger model achieves better performance but requires 3× more compute. Most practitioners use BERT-base because it offers a good balance of quality and efficiency.
def count_bert_parameters(
layers, hidden_size, attention_heads, ff_size, vocab_size=30522
):
"""
Count parameters in a BERT model.
"""
# Embedding layers
token_embeddings = vocab_size * hidden_size
position_embeddings = 512 * hidden_size
segment_embeddings = 2 * hidden_size
embedding_layernorm = 2 * hidden_size # gamma, beta
embeddings = (
token_embeddings
+ position_embeddings
+ segment_embeddings
+ embedding_layernorm
)
# Each transformer layer
# Multi-head attention: Q, K, V projections + output projection
attention = (
4 * hidden_size * hidden_size + 4 * hidden_size
) # weights + biases
# Feed-forward: two linear layers
ff = hidden_size * ff_size + ff_size # first layer
ff += ff_size * hidden_size + hidden_size # second layer
# Two layer norms per layer
layer_norms = 4 * hidden_size
layer_params = attention + ff + layer_norms
transformer_params = layers * layer_params
# Output heads
mlm_head = hidden_size * hidden_size + hidden_size # dense
mlm_head += hidden_size # layer norm
mlm_head += hidden_size * vocab_size + vocab_size # output projection
nsp_head = hidden_size * 2 + 2 # linear classifier
total = embeddings + transformer_params + mlm_head + nsp_head
return {
"embeddings": embeddings,
"transformer": transformer_params,
"mlm_head": mlm_head,
"nsp_head": nsp_head,
"total": total,
}Parameter breakdown (millions): Component BERT-base BERT-large -------------------------------------------------- embeddings 23.8M 31.8M transformer 85.1M 302.3M mlm_head 24.1M 32.3M nsp_head 0.0M 0.0M total 133.0M 366.4M
The embeddings account for a significant fraction of parameters due to the large vocabulary. The transformer layers dominate in the larger model. The prediction heads are relatively small.
