Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Tokenizer Training › A Complete Training Pipeline

Michael BrenndoerferUpdated March 21, 20263 min read

Language AI HandbookAll chapters

Part 5 of 67Subword Tokenization

Chapter 33Tokenizer Training › A Complete Training Pipeline

Reading section 8 of 13 · A Complete Training Pipeline · 3 minSection 8 of 13 · 3 min

Section 8 of 13 · 3 min read

Tokenizer Training › A Complete Training PipelineOne section · Sections in this chapter8 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The components we have assembled in this chapter combine into a reusable pipeline that you can adapt for any new domain. The key decisions are: what normalizer to use, which pre-tokenizer matches your target text format, what vocabulary size to target, and how to configure post-processing for your model architecture.

Let us combine all the components into a production-ready pipeline function. This function encapsulates the best-practice configuration for a GPT-style generative model: byte-level pre-tokenization for universal coverage, NFC normalization for consistent Unicode handling, and sequence boundary tokens for the model's expected input format:

In[39]:

Code

from tokenizers import Tokenizer
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
from tokenizers.models import BPE
from tokenizers.normalizers import NFC
from tokenizers.pre_tokenizers import ByteLevel
from tokenizers.processors import TemplateProcessing
from tokenizers.trainers import BpeTrainer


def train_production_tokenizer(
    corpus_iterator, vocab_size=32000, min_frequency=2, output_path=None
):
    """Train a production-ready BPE tokenizer with standard configuration."""

    # Byte-level BPE ensures universal character coverage
    tokenizer = Tokenizer(BPE(unk_token="<unk>"))

    # NFC normalization for consistent Unicode handling
    tokenizer.normalizer = NFC()

    # Byte-level pre-tokenization, GPT-2 style
    tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=True)

    # Decoder to properly reconstruct text from byte-level tokens
    tokenizer.decoder = ByteLevelDecoder()

    trainer = BpeTrainer(
        vocab_size=vocab_size,
        min_frequency=min_frequency,
        special_tokens=["<unk>", "<pad>", "<s>", "</s>", "<mask>"],
        show_progress=False,
    )

    tokenizer.train_from_iterator(corpus_iterator, trainer)

    # Post-processing adds sequence boundary tokens automatically
    tokenizer.post_processor = TemplateProcessing(
        single="<s> $A </s>",
        pair="<s> $A </s> $B </s>",
        special_tokens=[
            ("<s>", tokenizer.token_to_id("<s>")),
            ("</s>", tokenizer.token_to_id("</s>")),
        ],
    )

    if output_path:
        tokenizer.save(output_path)

    return tokenizer

Notice that the decoder is included alongside the encoder. This is important for production use: when the model generates token IDs and you want to convert them back to human-readable text, the decoder applies the inverse of the byte-level encoding to reconstruct the original characters. Without the decoder, you would get byte-level artifact characters in your output.

In[40]:

Code

sample_corpus = [
    "This is a sample corpus for training.",
    "It contains multiple sentences.",
    "The tokenizer will learn subword units from this text.",
    "Larger corpora produce better vocabularies.",
] * 100

production_tokenizer = train_production_tokenizer(
    corpus_iterator=iter(sample_corpus), vocab_size=1000, min_frequency=2
)

Out[41]:

Console

Production tokenizer output:
------------------------------------------------------------
Input:  'Hello, world!'
Tokens: ['<s>', 'Ġ', '<unk>', 'e', 'l', 'l', 'o', '<unk>', 'Ġ', 'w', 'or', 'l', 'd', '<unk>', '</s>']
IDs:    [2, 32, 0, 13, 19, 19, 22, 0, 32, 29, 33, 19, 12, 0, 3]

Input:  'This is a test sentence.'
Tokens: ['<s>', 'ĠThis', 'Ġis', 'Ġa', 'Ġt', 'es', 't', 'Ġsenten', 'ce', '.', '</s>']
IDs:    [2, 119, 88, 87, 34, 61, 26, 111, 45, 5, 3]

Input:  'Tokenization is important for NLP.'
Tokens: ['<s>', 'Ġ', 'T', 'oken', 'iz', 'a', 't', 'i', 'o', 'n', 'Ġis', 'Ġ', 'i', 'm', 'p', 'or', 't', 'a', 'nt', 'Ġfor', 'Ġ', '<unk>', 'L', '<unk>', '.', '</s>']
IDs:    [2, 32, 8, 75, 67, 9, 26, 17, 22, 21, 88, 32, 17, 20, 23, 33, 26, 9, 72, 116, 32, 0, 7, 0, 5, 3]

Each sequence begins with token ID 2 (<s>) and ends with token ID 3 (</s>), matching the expected format for sequence-to-sequence models. The byte-level encoding handles punctuation and spaces cleanly, with the Ġ prefix character marking word-initial spaces. This is the same convention used by GPT-2 and RoBERTa, which means your tokenizer is compatible with a wide range of existing architectures.