SentencePiece › Multilingual Vocabulary Allocation

When training a multilingual SentencePiece model, one of the most important design decisions is how to allocate vocabulary capacity across languages. If you train on a corpus with 90% English and 10% Chinese, BPE will allocate most of its vocabulary to English subwords because they are statistically dominant. The Chinese script, which uses a much larger character set, may end up with poor coverage.

SentencePiece addresses this through two mechanisms. The character_coverage parameter controls which characters receive explicit vocabulary entries, but does not directly control how vocabulary is allocated across languages. For multilingual models, practitioners often use a technique called language sampling: they oversample low-resource languages during the training corpus preparation phase so that the tokenizer sees approximately equal amounts of each language. Google's mT5 model used this approach with a temperature parameter to control how aggressively low-resource languages were upsampled.

The byte_fallback mechanism provides a safety net. Any character that did not make it into the vocabulary due to limited coverage can still be represented as a sequence of UTF-8 byte tokens. A Chinese character encoded as byte fallback tokens requires 3 bytes and therefore 3 tokens, which is less efficient than a dedicated vocabulary entry but much better than mapping to a single unknown token and losing all character identity.

You can observe this trade-off empirically. A tokenizer trained primarily on English that uses byte fallback for Chinese will produce very long token sequences for Chinese text, because most Chinese characters are not in the vocabulary and each requires 3 byte tokens. A tokenizer trained on balanced multilingual data will produce much shorter sequences for Chinese because the most common characters have dedicated vocabulary entries. For applications that need to process Chinese text efficiently, training a balanced multilingual tokenizer or a Chinese-specific tokenizer is worth the engineering investment.