Unigram Language Model Tokenization › Summary

The unigram language model tokenization algorithm takes a top-down, probabilistic approach to building subword vocabularies. Rather than constructing a vocabulary through merges (as in BPE and WordPiece), it starts with a large candidate vocabulary and prunes it using the EM algorithm, retaining tokens that maximize the corpus log-likelihood. The result is a vocabulary where every token earns its place by improving the model's ability to compress the training data.

The key takeaways are:

• Probabilistic foundation: Every token in the vocabulary has an associated probability, which allows computing a probability distribution over all possible segmentations of any word. This is the central conceptual contribution that distinguishes the algorithm from BPE and WordPiece.
• EM training: The algorithm alternates between computing expected token counts (E-step) and updating probabilities (M-step), then prunes the lowest-impact tokens until reaching the target vocabulary size. Each EM iteration is guaranteed to improve the corpus log-likelihood. This provides a reliable convergence signal.
• Viterbi decoding: At inference time, dynamic programming efficiently finds the maximum-probability segmentation of any word in $O(n^2)$ time, making inference fast despite the complexity of the underlying probability model.
• Subword regularization: By sampling from the segmentation distribution instead of always taking the best, downstream models are exposed to diverse tokenizations during training, improving robustness to tokenization variation. The temperature parameter $\alpha$ controls the tradeoff between consistency and diversity.
• SentencePiece integration: The SentencePiece library provides a production-ready implementation that handles raw Unicode text directly, uses a whitespace marker to encode word boundaries, and supports multilingual vocabulary training across more than 100 languages.
• Limitations to keep in mind: Training cost is higher than BPE, the independence assumption means absolute probabilities are not linguistically accurate, and the pruning process may reach local rather than global optima. These limitations are acceptable for most applications but worth understanding when diagnosing unexpected behavior.

Building on the vocabulary-problem motivation from earlier in this part and the BPE and WordPiece algorithms we have already covered, the unigram LM completes our survey of the core subword tokenization approaches. Each algorithm represents a different philosophy: BPE is simple and fast (frequency-driven merges), WordPiece is likelihood-aware (merge selection based on probability), and unigram LM is fully probabilistic (top-down pruning with a principled objective). The next chapter covers SentencePiece in depth, exploring how it unifies BPE and unigram LM training under a single framework and handles the practical challenges of multilingual tokenization at production scale.