BERT Pre-training › Pre-training Data
Language AI HandbookAll chapters BERT Pre-training › Pre-training DataOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT's knowledge comes from text. The original model trained on two massive corpora: the BooksCorpus and English Wikipedia. Together, these sources provide around 3.3 billion words of diverse, high-quality text.
A collection of approximately 11,000 unpublished books spanning multiple genres, totaling around 800 million words. Originally compiled for book-to-movie alignment research, it provides long-form narrative text with coherent multi-sentence structure.
The BooksCorpus contributes long-form narrative text with coherent story structure. Each book maintains consistent style, character references, and thematic development across thousands of sentences. This teaches the model long-range dependencies that isolated sentences cannot provide.
Wikipedia adds encyclopedic knowledge. Its articles cover many specialized domains, from historical and scientific topics to biographies. The text is well-edited, factually dense, and structurally organized with clear section boundaries. From Wikipedia, BERT learns factual associations, entity relationships, and formal writing conventions.
The combination matters. Fiction teaches narrative flow, dialogue patterns, and emotional language. Non-fiction teaches factual structure, technical vocabulary, and logical argumentation. Together, they produce representations that generalize across domains.
Document Structure
Pre-training data must preserve document boundaries because one of BERT's objectives requires sampling sentence pairs from the same document. Each document is processed as a sequence of sentences, maintaining the original order.
The data pipeline typically proceeds as follows:
- Document collection: Gather raw text organized by document (books, articles, pages)
- Sentence segmentation: Split documents into individual sentences using punctuation and linguistic rules
- Tokenization: Convert sentences to WordPiece token sequences
- Pair construction: Create training examples by combining sentence pairs
For Wikipedia, each article forms a document. For BooksCorpus, each book forms a document. Sentence boundaries are detected using simple heuristics (periods followed by spaces and capital letters) combined with special handling for abbreviations and quotations.
import re
def segment_sentences(text):
"""
Simple sentence segmentation.
Production systems use more sophisticated methods.
"""
# Split on sentence-ending punctuation followed by space and capital
pattern = r"(?<=[.!?])\s+(?=[A-Z])"
sentences = re.split(pattern, text)
# Filter empty strings and strip whitespace
return [s.strip() for s in sentences if s.strip()]
# Example document
sample_doc = """
The transformer architecture changed NLP. It introduced self-attention as
the primary mechanism. Previous models relied heavily on recurrence.
Attention allows parallel processing of sequences. This enables much
faster training on modern hardware.
"""
sentences = segment_sentences(sample_doc)Document segmented into 5 sentences: 1. The transformer architecture changed NLP. 2. It introduced self-attention as the primary mechanism. 3. Previous models relied heavily on recurrence. 4. Attention allows parallel processing of sequences. 5. This enables much faster training on modern hardware.
Real preprocessing pipelines need separate rules for abbreviations ("Dr.", "U.S.") and URLs. They also need to preserve quotation boundaries and numbered lists. BERT's original preprocessing used a combination of sentence boundary detection libraries and manual filtering.
Creating Training Examples
Each training example for BERT consists of two segments (which we call sentence A and sentence B) joined with special tokens. The format is:
[CLS] sentence A tokens [SEP] sentence B tokens [SEP]
where [CLS] is a special classification token and [SEP] marks segment boundaries.
The two sentences are drawn from the same document. Half the time, sentence B immediately follows sentence A in the original text (a "positive" pair). Half the time, sentence B is a random sentence from a different location in the corpus (a "negative" pair). This sampling strategy creates the data for the Next Sentence Prediction task.
import random
def create_training_pairs(documents, positive_ratio=0.5):
"""
Create sentence pairs for BERT pre-training.
Args:
documents: List of documents, each a list of sentences
positive_ratio: Fraction of pairs from consecutive sentences
Returns:
List of (sentence_a, sentence_b, is_next) tuples
"""
pairs = []
all_sentences = [sent for doc in documents for sent in doc]
for doc in documents:
for i in range(len(doc) - 1):
sentence_a = doc[i]
if random.random() < positive_ratio:
# Positive: actual next sentence
sentence_b = doc[i + 1]
is_next = True
else:
# Negative: random sentence from corpus
sentence_b = random.choice(all_sentences)
is_next = False
pairs.append((sentence_a, sentence_b, is_next))
return pairs# Create sample documents
sample_documents = [
[
"The cat sat on the mat.",
"It was a sunny afternoon.",
"Birds sang in the nearby trees.",
],
[
"Machine learning transforms data into predictions.",
"Neural networks learn hierarchical representations.",
"Deep learning requires substantial compute.",
],
]
training_pairs = create_training_pairs(sample_documents)Sample training pairs: 1. [NotNext] A: The cat sat on the mat. B: The cat sat on the mat. 2. [NotNext] A: It was a sunny afternoon. B: It was a sunny afternoon. 3. [IsNext] A: Machine learning transforms data into predictions. B: Neural networks learn hierarchical representations. 4. [NotNext] A: Neural networks learn hierarchical representations. B: Deep learning requires substantial compute.
The example shows how the sampling works. Consecutive sentence pairs teach the model about discourse coherence. Random pairs teach it to distinguish coherent text from jumbled fragments. This binary classification signal, combined with masked token prediction, gives BERT a richer understanding than masking alone would provide.
Sequence Length and Packing
BERT processes fixed-length sequences, typically 512 tokens. Most sentence pairs are shorter than this limit, so multiple pairs can be packed into a single sequence to maximize GPU utilization.
The packing strategy concatenates sentence pairs until the sequence reaches the maximum length:
[CLS] pair1_A [SEP] pair1_B [SEP] [CLS] pair2_A [SEP] pair2_B [SEP] ...
However, the original BERT paper uses a simpler approach: each training example contains exactly one sentence pair, padded to the maximum length. This wastes some compute on padding tokens but simplifies the training pipeline.
A compromise used in the original BERT paper is to sample 90% of training examples with short sequences (128 tokens) and 10% with the full length (512 tokens). Short sequences speed up initial training when the model is learning basic patterns, while long sequences in the final phase provide the context needed for complex relationships.