Tokenizer Training › Key Parameters
Language AI HandbookAll chapters Tokenizer Training › Key ParametersOne section · Sections in this chapter12 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The following parameters are the most important when training tokenizers with the HuggingFace tokenizers library. Understanding what each parameter controls helps you make informed choices rather than relying on defaults.
BpeTrainer, WordPieceTrainer, and UnigramTrainer
These trainer classes share the same core parameters for controlling vocabulary learning. The parameters govern how the training algorithm decides which character sequences are frequent enough to earn vocabulary entries:
| Parameter | Description | Typical Values |
|---|---|---|
vocab_size | Target vocabulary size including special tokens. Larger values produce shorter sequences but require more memory. | 8,000-100,000 |
min_frequency | Minimum number of times a token must appear to be included. Higher values produce cleaner vocabularies but may miss important rare terms. | 2-5 |
special_tokens | Tokens guaranteed to be in vocabulary with fixed IDs. Order matters: the first token gets ID 0. | ["[UNK]", "[PAD]", "[CLS]", "[SEP]", "[MASK]"] |
show_progress | Whether to display a progress bar during training. | True or False |
Pre-tokenizers
The pre-tokenizer determines how raw text is split into word-level chunks before the subword model processes each chunk. The choice affects whether punctuation is attached to adjacent words, how whitespace is handled, and whether the subword model ever sees characters outside the ASCII range directly.
| Pre-tokenizer | When to Use |
|---|---|
Whitespace() | Simple whitespace splitting. Good for quick experiments and whitespace-delimited text. |
ByteLevel(add_prefix_space=True) | GPT-2 style, ensures universal character coverage. Best for production models. |
Metaspace() | SentencePiece style with the ▁ (U+2581) marker for word boundaries. Good for multilingual use. |
Normalizers
Normalizers transform the raw input string before any splitting occurs. Because normalization happens before vocabulary lookup, normalization decisions affect which vocabulary entries are created during training and whether they match at inference time. Training and inference must use identical normalizers.
| Normalizer | Effect |
|---|---|
NFC() / NFD() / NFKC() / NFKD() | Unicode normalization forms. NFC preserves characters; NFKC applies compatibility normalization. |
Lowercase() | Converts all text to lowercase. Reduces vocabulary but loses case information. |
StripAccents() | Removes accent marks. Useful for ASCII-focused vocabularies. |
Sequence([...]) | Chains multiple normalizers in order. |
Post-processors
Post-processors add special tokens and format the final output after the subword model has produced its segmentation. The post-processor is configured at training time but applied at inference time, so it must match the format your model expects.
| Post-processor | Purpose |
|---|---|
TemplateProcessing(single="[CLS] $A [SEP]", ...) | Adds special tokens around sequences. Configure for BERT-style or GPT-style formats. |
ByteLevel(trim_offsets=True) | Required when using byte-level pre-tokenization to properly handle token boundaries. |