SentencePiece › Summary

SentencePiece eliminates pretokenization by treating raw text as a sequence of Unicode characters and handling whitespace through the ▁ prefix encoding. This single design decision enables language-agnostic tokenization: the same code, the same algorithm, and the same vocabulary structure work for English, Chinese, Arabic, or any language without modification.

The ▁ prefix is the key mechanism. By replacing spaces with a visible character before applying the subword algorithm, SentencePiece preserves word boundary information in a form that the algorithm can learn from, without requiring any language-specific rules about what constitutes a word. The encoding is perfectly invertible, so you can always recover the original text from the tokens.

The library provides two algorithms: BPE, which builds vocabulary bottom-up by greedily merging the most frequent adjacent pairs at each step, and the unigram language model, which prunes top-down from a large initial vocabulary using EM. BPE is deterministic and fast at inference; unigram enables subword regularization, which improves downstream model quality for sequence-to-sequence tasks by training the model to handle multiple valid segmentations of the same text.

Key practical takeaways:

• The ▁ prefix on tokens like "▁the" marks word starts; tokens without ▁ are continuations
• Vocabulary size of 32,000 is a common production choice, balancing compression against embedding table size
• Set character_coverage=1.0 and byte_fallback=True for multilingual models
• The unigram model's probabilistic segmentation enables subword regularization as data augmentation
• Saved .model files are portable: load them with sentencepiece in Python, C++, or any other environment

Many of the models you will encounter in the rest of this book, including T5, LLaMA, and the multilingual transformer families, use SentencePiece for tokenization. Understanding how it works under the hood will help you interpret tokenization behavior when you encounter unexpected splits, make informed choices about vocabulary size and algorithm selection when training your own models, and debug tokenization issues when they arise in production. The principles here, treating whitespace as data, learning structure from frequency, and separating the tokenizer from the model, will recur throughout the remaining chapters as we build increasingly powerful language systems.