Tokenizer Training › Worked Example: Tracing the Full Pipeline
Language AI HandbookAll chapters Tokenizer Training › Worked Example: Tracing the Full PipelineOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To solidify the concepts from this chapter, let us trace a single string through the complete tokenization pipeline step by step. We will observe what each component does and see how the vocabulary from training shapes the final output.
Consider the input string "Transformers learn language patterns." being processed by a BPE tokenizer with whitespace pre-tokenization, lowercase normalization, and no post-processing. We will trace each stage manually.
Stage 1: Normalization. The normalizer applies lowercase conversion, transforming "Transformers learn language patterns." to "transformers learn language patterns.". If NFD and StripAccents are also applied, any accented characters would be decomposed and stripped, but this input has none.
Stage 2: Pre-tokenization. The whitespace pre-tokenizer splits on spaces, producing the word list: ["transformers", "learn", "language", "patterns."]. Note that punctuation attached to words (the period in "patterns.") stays attached in whitespace splitting. Byte-level pre-tokenization would handle this differently, treating every character as a separate unit before the subword model runs.
Stage 3: BPE segmentation. Each word is independently segmented by the BPE model. BPE starts with character-level representation and applies learned merges. If the vocabulary contains "tr", "an", "sfo", "rme", "rs", "transformers" as merged tokens, the word is segmented accordingly. The exact segmentation depends on which merges were learned during training, which depends on the training corpus.
Stage 4: Post-processing. If a post-processor is configured, special tokens are added. With single="[CLS] $A [SEP]", the final token sequence becomes ["[CLS]", ...tokens..., "[SEP]"] with the special tokens prepended and appended.
Stage 5: ID lookup. Each token string is looked up in the vocabulary to retrieve its integer ID. The result is a list of integers that the model will process.
This five-stage pipeline is what happens every time you call tokenizer.encode(text). Understanding each stage helps you diagnose tokenization issues: if your outputs look strange, you can check whether the problem is in normalization (characters being transformed unexpectedly), pre-tokenization (words being split at unexpected boundaries), or the subword model itself (vocabulary gaps causing excessive fragmentation).