Tokenizer Training › Corpus Preparation
Language AI HandbookAll chapters Tokenizer Training › Corpus PreparationOne section · Sections in this chapter2 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The quality of your tokenizer depends entirely on the quality of your training corpus. A tokenizer learns which character sequences are common enough to become tokens. If your corpus does not represent your target domain, the tokenizer will produce suboptimal splits at inference time. This dependency is not something you can compensate for at the model training stage. Once the tokenizer is trained and its vocabulary fixed, the quality of that vocabulary is locked in.
Think of corpus preparation as deciding what your tokenizer gets to study. A student who studies exclusively history textbooks will not know how to read a chemistry paper. A tokenizer trained exclusively on news articles will not know that dataclass, isinstance, and -> are meaningful units in Python. The corpus is the curriculum, and the vocabulary that emerges reflects exactly what was in that curriculum.
The relationship between corpus choice and learned vocabulary affects correctness, not only optimization. A tokenizer trained on news text will fragment Python code into individual characters because identifiers like enumerate, isinstance, and dataclass never appeared in its training data. The model downstream then has to reconstruct meaning from pieces that do not correspond to semantic units. Every step of downstream learning is made harder by this fragmentation.
The collection of text used to learn tokenizer vocabulary. The corpus determines which subwords are frequent enough to become tokens, so it should represent the text the tokenizer will process at inference time. For a general-purpose model, this corpus typically spans billions of tokens drawn from diverse web content, books, and code. For a domain-specific model, a smaller but carefully curated corpus is often more effective than a large general-purpose one.
What Makes a Good Training Corpus?
A well-chosen corpus has three properties that are worth examining carefully, because they affect the quality of the resulting vocabulary in distinct and predictable ways.
Representative text is the most important property. The corpus should match the distribution of text you will tokenize in production. If your model will process medical records, train your tokenizer on medical records. If your model will process legal contracts, train on legal contracts. The intuition here is straightforward: BPE merges, WordPiece scoring, and Unigram probability estimation all depend on raw token frequencies. Tokens that appear frequently in the training corpus will be merged into longer units. Tokens that appear rarely will be left as character fragments. If your target domain uses different vocabulary than your training corpus, important domain terms will be systematically fragmented.
Large enough corpus is needed for frequency statistics to be meaningful. For general-purpose tokenizers, this means billions of tokens. For domain-specific tokenizers, millions often suffice. The threshold depends on vocabulary size: if you want 32,000 vocabulary entries, you need enough data that 32,000 different character sequences are distinguishably frequent. A corpus with only 10,000 sentences may not provide reliable frequency statistics even for a modest 1,000-token vocabulary.
Clean text is essential because noise in the corpus becomes noise in the vocabulary. HTML artifacts, encoding errors, and garbage text waste vocabulary slots on useless tokens. A tokenizer trained on HTML will learn to keep </div> and & as single tokens, which serves little purpose for a model that needs to understand prose. Cleaning the corpus before training ensures vocabulary slots go to meaningful linguistic units.
Let us examine how corpus choice affects the learned vocabulary:
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.trainers import BpeTrainer
def train_tokenizer_on_corpus(texts, vocab_size=1000):
"""Train a BPE tokenizer on the given texts."""
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(
vocab_size=vocab_size,
special_tokens=["[UNK]", "[PAD]", "[CLS]", "[SEP]", "[MASK]"],
show_progress=False,
)
tokenizer.train_from_iterator(texts, trainer)
return tokenizer
# Two different corpora
general_corpus = [
"The cat sat on the mat.",
"Dogs are loyal companions.",
"Birds fly south for winter.",
"The weather is nice today.",
"She walked to the store.",
] * 100 # Repeat for statistical significance
code_corpus = [
"def calculate_sum(a, b): return a + b",
"for i in range(10): print(i)",
"if x > 0: return True else: return False",
"import numpy as np; import pandas as pd",
"class DataLoader: def __init__(self): pass",
] * 100
general_tokenizer = train_tokenizer_on_corpus(general_corpus, vocab_size=200)
code_tokenizer = train_tokenizer_on_corpus(code_corpus, vocab_size=200)Tokenizing code with different tokenizers:
--------------------------------------------------
Input: 'def load_data(): return None'
General tokenizer (21 tokens):
['d', 'e', 'f', 'lo', 'a', 'd', '[UNK]', 'd', 'at', 'a', '[UNK]', '[UNK]', '[UNK]', 're', 't', 'u', 'r', 'n', '[UNK]', 'on', 'e']
Code tokenizer (16 tokens):
['def', 'l', 'o', 'a', 'd', '_', 'd', 'at', 'a', '(', '):', 'return', '[UNK]', 'o', 'n', 'e']The difference is striking. The general-purpose tokenizer, trained on natural language, fragments the code into many small pieces because it never learned that def, return, or None are meaningful units. The code tokenizer recognizes these as complete tokens, producing a more compact and semantically meaningful representation. This matters for the downstream model in two ways: shorter sequences are processed faster by attention, and tokens that correspond to meaningful units are easier to learn associations for.
Let us measure this difference systematically across several code snippets:
Token count comparison across code snippets: ----------------------------------------------------------------- Code Snippet | General | Code | Reduction ----------------------------------------------------------------- def sum(a, b): | 12 | 8 | 33% return x + y | 8 | 4 | 50% for i in range(10): | 13 | 7 | 46% import numpy as np | 13 | 4 | 69% class Model: | 11 | 6 | 45%
Corpus choice has a dramatic effect on tokenization efficiency. For code, the domain-specific tokenizer reduces token counts by 50-70%, which translates directly to faster training and inference. Beyond speed, the semantic alignment matters: a model learning from code representations where enumerate is a single token can build a clean association between that token and its semantics, rather than having to learn that the fragments en, um, er, ate collectively mean something specific.
Preprocessing for Tokenizer Training
Before training, you typically preprocess the corpus to remove noise and normalize text. The specific steps depend on your domain, but several apply broadly regardless of what you are tokenizing.
Preprocessing must produce consistent text, not simply clean text. Two strings that mean the same thing but differ in whitespace, Unicode encoding, or HTML markup should look the same to the tokenizer. If they do not, the tokenizer may learn separate statistics for each surface form, diluting the frequency counts that determine vocabulary membership.
import re
import unicodedata
def preprocess_for_tokenizer(text):
"""Clean text for tokenizer training."""
# Remove HTML tags
text = re.sub(r"<[^>]+>", "", text)
# Normalize whitespace
text = re.sub(r"\s+", " ", text)
# Remove control characters (except newlines and tabs)
text = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]", "", text)
# Normalize Unicode to composed form
text = unicodedata.normalize("NFC", text)
return text.strip()
# Example of preprocessing
raw_text = """<p>Hello world!</p>
Some text with extra spaces
And control chars: \x00\x01\x02"""
cleaned = preprocess_for_tokenizer(raw_text)Before preprocessing: '<p>Hello world!</p>\nSome text with extra spaces\nAnd control chars: \x00\x01\x02' After preprocessing: 'Hello world! Some text with extra spaces And control chars:'
The preprocessing removed the HTML tags, collapsed multiple spaces into single spaces, and stripped the control characters. Notice how <p>Hello world!</p> became just Hello world!. This cleanup ensures your vocabulary contains meaningful tokens rather than HTML fragments or encoding artifacts.
Two preprocessing decisions deserve particular attention because they have large downstream effects. The first is lowercasing: converting all text to lowercase reduces vocabulary size by collapsing case variants, but costs the model its ability to distinguish proper nouns from common words. For a general text model, this is often a worthwhile tradeoff. For a model that needs to distinguish "Apple" (the company) from "apple" (the fruit), it is not. The second decision is Unicode normalization. NFC (composed form) is safest for most use cases because it preserves characters while ensuring consistent representation of accented characters, for instance ensuring that "é" is always stored as a single code point rather than as "e" followed by a combining accent. NFKC is more aggressive, normalizing compatibility variants like fullwidth characters, which can help multilingual tokenizers but may alter text in unexpected ways, such as converting the ligature "fi" into two separate characters.
A third decision worth considering carefully is sentence segmentation. Many tokenizer trainers expect one sentence or document per line. If you feed entire multi-paragraph documents as single training examples, the tokenizer may include newline characters in its learned merges. If you strip all newlines before training, you lose the information about sentence and paragraph boundaries. A common practice is to use sentence-level segmentation for the training corpus. This ensures each training example is a coherent unit of text.