BERT Pre-training › Summary
Michael BrenndoerferUpdated July 19, 20251 min read
Language AI HandbookAll chapters Part 24 of 67BERT and Variants
BERT Pre-training › SummaryOne section · Sections in this chapter11 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT's pre-training combines two complementary objectives to learn rich language representations:
- Training data from BooksCorpus and Wikipedia provides diverse, high-quality text totaling 3.3 billion words
- Masked Language Modeling forces bidirectional context understanding by predicting randomly masked tokens (15% mask rate, 80-10-10 corruption strategy)
- Next Sentence Prediction adds document-level signal by classifying whether sentence pairs are consecutive (though its value is debated)
- Hyperparameters include AdamW optimizer with linear warmup and decay, 256 batch size, and two-phase sequence length (128 then 512)
- Training duration spans 1 million steps over approximately 4 days for BERT-base, processing roughly 40 billion tokens
- The combined loss equally weights MLM and NSP, allowing both objectives to shape the shared representations
The next chapter explores fine-tuning: how to adapt these pre-trained representations to specific downstream tasks.