Unigram Language Model Tokenization › SentencePiece Unigram in Production

SentencePiece's unigram mode is used in production in several important ways that go beyond what simple demonstrations reveal. Understanding these production characteristics helps you use the library effectively in real systems.

The algorithm operates directly on raw Unicode text without requiring any pre-tokenization step (unlike BPE implementations that often require whitespace pre-splitting). This is a significant advantage for multilingual text, where the boundary between "words" may not be meaningful, such as in Chinese, Japanese, or Thai. In these languages, there are no spaces between words, so any tokenizer that requires word boundaries as input must first run a separate word segmentation step. SentencePiece treats the entire character sequence as its input, learning subword patterns that respect the linguistic structure of each language automatically.

SentencePiece introduces a special whitespace marker, the character ▁ (U+2581, a lower one-eighth block), to encode word boundaries within the token stream. A token beginning a new word is prefixed with this character, while continuation tokens within a word have no prefix. This allows the tokenizer to reconstruct the original whitespace from the token sequence without storing it separately. For example, the sentence "hello world" might tokenize as ["▁hello", "▁world"], and you can recover the original text with spaces by stripping the ▁ markers and inserting spaces before each one.

For multilingual models like mBART, mT5, and XLM-RoBERTa, SentencePiece unigram mode trains a single shared tokenizer across all languages simultaneously. By training on multilingual data with appropriate character coverage settings, the resulting vocabulary naturally allocates token capacity to each language proportional to its representation in training data. Languages with simpler morphology (like English) tend to get many whole-word tokens, while languages with complex morphology (like Finnish or Turkish) get more subword tokens that compose into the rich inflectional paradigms those languages use.

The whitespace handling and language-agnostic character coverage combine to make SentencePiece unigram mode the preferred choice for multilingual NLP. When you see a model like mBART described as using "SentencePiece with 250,000 vocabulary," that vocabulary supports over 100 languages with appropriate coverage of their alphabets and common morphological patterns.