BERT Pre-training › Pre-training Duration and Convergence

Michael BrenndoerferUpdated July 19, 20254 min read
Language AI HandbookAll chapters
Part 24 of 67BERT and Variants
Chapter 170BERT Pre-training › Pre-training Duration and Convergence
Reading section 7 of 12 · Pre-training Duration and Convergence · 4 minSection 7 of 12 · 4 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 170 of 513

Section 7 of 12 · 4 min read

BERT Pre-training › Pre-training Duration and ConvergenceOne section · Sections in this chapter7 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

How do you know when pre-training is done? BERT trained for 1 million steps, but the loss curve provides more insight than a fixed number.

Loss Curves

Both MLM and NSP losses decrease rapidly in early training, then gradually level off. The MLM loss typically drops from around 10 (random baseline) to 1.5-2.0. The NSP loss drops from 0.69 (random) to below 0.1.

Out[36]:
Visualization
Line plot showing MLM loss starting around 10 and dropping to 2, NSP loss dropping quickly to near 0.
Simulated BERT pre-training loss curves. MLM loss dominates early training and continues improving throughout. NSP loss converges quickly as the task is easier. The total loss is the sum of both.

The curves show that NSP converges much faster than MLM. After about 100K steps, the NSP loss is near its final value. MLM continues improving throughout training, though with diminishing returns after 500K steps.

Validation Metrics

During pre-training, the primary metrics are MLM accuracy (fraction of masked tokens correctly predicted) and NSP accuracy. These provide intuition about how well the model is learning.

Typical final values for BERT-base:

  • MLM accuracy: 60-65%
  • NSP accuracy: 97-98%

The high NSP accuracy reflects that the task is relatively easy. The MLM accuracy of 60% might seem low, but remember the model is predicting from a 30,000-token vocabulary with only surrounding context. Many masked positions have multiple plausible answers.

Out[37]:
Visualization
Line plot showing MLM accuracy rising to 60% and NSP accuracy rising quickly to 97%.
Simulated validation accuracy during BERT pre-training. NSP accuracy rises quickly to near-perfect levels (the task is relatively easy), while MLM accuracy improves more gradually, plateauing around 60-65% due to the inherent difficulty of vocabulary prediction.

When to Stop

Pre-training should stop when:

  1. Loss plateaus: The MLM loss has stopped decreasing for many steps
  2. Downstream performance peaks: Validation on downstream tasks shows no improvement
  3. Compute budget exhausted: The allocated resources are fully used

In practice, most BERT variants train for a fixed number of steps (1M for original BERT, 100K for RoBERTa's larger batches) determined empirically. Longer training generally helps until the model begins overfitting to the pre-training corpus.