Tokenizer Training › Training with HuggingFace Tokenizers
Language AI HandbookAll chapters Tokenizer Training › Training with HuggingFace TokenizersOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The HuggingFace tokenizers library provides a fast, flexible framework for tokenizer training. It supports BPE, WordPiece, and Unigram models with customizable pre-tokenization, normalization, and post-processing. The library is written in Rust for performance, but exposes a clean Python API that mirrors the conceptual structure of the tokenization pipeline.
The design philosophy of the library is worth understanding because it shapes how you think about tokenizer configuration. Rather than hardcoding the full tokenization pipeline, the library decomposes it into four independent components that you can mix and match. This modularity lets you reproduce BERT's tokenization exactly, or GPT-2's, or create a novel combination suited to your task.
The Tokenizer Pipeline
A HuggingFace tokenizer has four components that process text in sequence. Understanding each component and its role helps you make informed configuration choices rather than copying settings without understanding their effects.
Normalizer: Transforms raw text before tokenization. This is where lowercasing, Unicode normalization, and accent stripping happen. Normalization is applied to the full text string before it is split into any units, which means normalization decisions affect how the tokenizer's vocabulary aligns with raw text.
Pre-tokenizer: Splits text into words or word-like units before the subword model runs. Whitespace splitting is the simplest pre-tokenizer: it splits on spaces and newlines, treating each resulting chunk as a word to be further processed by the subword model. Byte-level pre-tokenization, used by GPT-2, maps every character to a unique byte token before splitting. This ensures that the subword model always operates on sequences of printable characters regardless of the input.
Model: The core subword algorithm (BPE, WordPiece, or Unigram) that takes the pre-tokenized words and splits each one into subword tokens. This is where the statistical learning from the training corpus is encoded. The model stores the vocabulary and the rules for segmentation.
Post-processor: Adds special tokens and formats the final output. This is where [CLS] and [SEP] tokens are inserted for BERT-style models, or <s> and </s> for sequence-to-sequence models. Post-processing is the final step before the tokenizer returns token IDs to the caller.
This modular design lets you mix and match components. You might use byte-level pre-tokenization (from GPT-2) with WordPiece scoring (from BERT) if your use case calls for it. The components are independent, and their effects compose predictably.
Let us build a complete tokenizer with all four components configured:
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.normalizers import NFD, Lowercase, Sequence, StripAccents
from tokenizers.pre_tokenizers import ByteLevel
# Initialize tokenizer with BPE model
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
# Configure normalizer: NFD decomposition, lowercase, strip accents
tokenizer.normalizer = Sequence([NFD(), Lowercase(), StripAccents()])
# Use byte-level pre-tokenization (like GPT-2)
tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=True)The normalizer chain processes text before any tokenization occurs. NFD normalization decomposes characters into base characters and combining marks, making it straightforward to strip accents consistently. The ordering matters: you must decompose with NFD before stripping accents, because NFC composition would re-merge the accent back into the base character. Lowercasing reduces vocabulary size by treating "The" and "the" as the same token.
training_texts = [
"The quick brown fox jumps over the lazy dog.",
"Machine learning is transforming how we process language.",
"Natural language processing enables many applications.",
"Deep learning models require large amounts of training data.",
"Attention mechanisms allow models to focus on relevant parts of the input.",
"Transformers have become the dominant architecture for NLP tasks.",
"Pre-training on large corpora improves downstream task performance.",
"Fine-tuning adapts pre-trained models to specific domains.",
] * 100
trainer = BpeTrainer(
vocab_size=1000,
min_frequency=2,
special_tokens=["[UNK]", "[PAD]", "[CLS]", "[SEP]", "[MASK]"],
show_progress=False,
)
tokenizer.train_from_iterator(training_texts, trainer)Trained vocabulary size: 261
First 10 tokens (special + byte-level base):
0: '[UNK]'
1: '[PAD]'
2: '[CLS]'
3: '[SEP]'
4: '[MASK]'
5: '-'
6: '.'
7: 'a'
8: 'b'
9: 'c'
Sample learned merge tokens:The vocabulary structure reveals the tokenizer's architecture. Special tokens occupy the first slots with fixed IDs 0-4. The 256 byte-level base tokens follow. This ensures any character can be represented. The remaining slots contain merged tokens: progressively longer sequences that BPE identified as frequent in the training corpus. This byte-level encoding guarantees the tokenizer can handle any input, even characters not seen during training, by falling back to individual byte tokens rather than the unknown token [UNK].
The histogram reveals the vocabulary's layered structure. Single-character tokens form the base layer, guaranteeing universal coverage. Most merged tokens are 2-5 characters, representing common morphemes like "ing", "tion", and "pre". Longer tokens capture frequently occurring words that appeared often enough in the corpus to earn dedicated vocabulary slots. This distribution is characteristic of well-trained BPE vocabularies on English text. A vocabulary with an unusual distribution (for instance, dominated by very long tokens) often signals a training corpus that was too small or too repetitive.
Adding Post-Processing
Post-processing adds special tokens that models expect. This step is where your tokenizer's output format is configured to match the input format your model was designed for. Different model architectures expect different special token conventions, and getting this right is essential for compatibility.
BERT-style models need [CLS] at the start and [SEP] between segments. The [CLS] token is an aggregate representation of the full sequence in BERT's design, and many later tasks read the model's prediction from the [CLS] position. The [SEP] token marks the boundary between two input segments when the model is processing sentence pairs, as in natural language inference tasks.
from tokenizers.processors import TemplateProcessing
tokenizer.post_processor = TemplateProcessing(
single="[CLS] $A [SEP]",
pair="[CLS] $A [SEP] $B [SEP]",
special_tokens=[
("[CLS]", tokenizer.token_to_id("[CLS]")),
("[SEP]", tokenizer.token_to_id("[SEP]")),
],
)Input: 'Hello world!' Tokens: ['[CLS]', 'Ġ', 'he', 'll', 'o', 'Ġ', 'w', 'o', 'r', 'l', 'd', '[UNK]', '[SEP]'] IDs: [2, 33, 51, 132, 21, 33, 29, 21, 24, 18, 10, 0, 3]
The post-processor automatically wraps the input with [CLS] and [SEP] tokens, matching the format expected by BERT and similar models. GPT-style models use <s> and </s> instead; you configure the template accordingly. Sequence-to-sequence models often use both, with different conventions for the encoder and decoder inputs.
Training Different Model Types
The tokenizers library supports three subword algorithms. Each has a different training procedure and produces different segmentations, but the API is consistent: you create a trainer for the corresponding algorithm and call train_from_iterator. Here is how to train each:
from tokenizers.models import Unigram, WordPiece
from tokenizers.trainers import UnigramTrainer, WordPieceTrainer
# BPE Tokenizer
bpe_tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
bpe_tokenizer.pre_tokenizer = Whitespace()
bpe_trainer = BpeTrainer(
vocab_size=500, special_tokens=["[UNK]", "[PAD]"], show_progress=False
)
bpe_tokenizer.train_from_iterator(training_texts, bpe_trainer)
# WordPiece Tokenizer
wp_tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))
wp_tokenizer.pre_tokenizer = Whitespace()
wp_trainer = WordPieceTrainer(vocab_size=500, special_tokens=["[UNK]", "[PAD]"])
wp_tokenizer.train_from_iterator(training_texts, wp_trainer)
# Unigram Tokenizer
unigram_tokenizer = Tokenizer(Unigram())
unigram_tokenizer.pre_tokenizer = Whitespace()
unigram_trainer = UnigramTrainer(
vocab_size=500, special_tokens=["[UNK]", "[PAD]"]
)
unigram_tokenizer.train_from_iterator(training_texts, unigram_trainer)Tokenizing: 'Transformers process language efficiently.' -------------------------------------------------- BPE : ['Transformers', 'process', 'language', 'e', 'f', 'fic', 'i', 'en', 't', 'l', 'y', '.'] WordPiece : ['Transformers', 'process', 'language', 'e', '##f', '##f', '##ic', '##i', '##e', '##nt', '##l', '##y', '.'] Unigram : ['T', 'ransform', 'e', 'r', 's', 'process', 'l', 'a', 'ng', 'u', 'a', 'g', 'e', 'e', 'f', 'f', 'ic', 'i', 'e', 'n', 't', 'l', 'y', '.']
The three algorithms produce noticeably different segmentations for the same input. BPE tends to produce longer common subwords through greedy merging of frequent pairs: it merges the most common adjacent pair at each step, building up longer tokens incrementally. WordPiece often shows the ## prefix for continuation tokens within words. This reflects its likelihood-based scoring: it adds a merge only when it improves the language model likelihood of the training data. Unigram may choose different boundaries based on global probability optimization: it starts with a large vocabulary and prunes tokens whose removal least degrades the overall probability of the training corpus.
In practice, the choice of algorithm matters less than vocabulary size and corpus quality. All three algorithms converge to similar vocabularies when trained on large enough data. The differences are most pronounced at small vocabulary sizes and on unusual input text. The real reason to prefer one algorithm over another is usually compatibility: if you are building a BERT-style model, use WordPiece. If you are building a GPT-style model, use BPE. If you are using SentencePiece, use Unigram.