Unigram Language Model Tokenization › Unigram vs. BPE: A Conceptual Comparison

The unigram LM and BPE approaches embody different philosophies about what makes a good tokenizer. Understanding both philosophies helps you reason about when to use each and what tradeoffs to expect.

BPE is constructive: it starts small and grows. The vocabulary begins as individual characters, and merges create new tokens by combining the most frequent existing pairs. The merge order is determined purely by co-occurrence frequency in the current vocabulary state. BPE is deterministic given the same corpus and hyperparameters, and it produces exactly one segmentation per word at inference time. The algorithm is simple enough to implement in a few dozen lines of Python and fast enough to train on large corpora in minutes.

The unigram LM is reductive: it starts large and shrinks. The vocabulary begins with many candidates, the EM algorithm assigns each a probability based on how much it contributes to explaining the training data, and pruning removes low-value tokens. At inference time, the model can produce multiple segmentations with associated probabilities. The algorithm requires understanding the EM framework, the Viterbi forward-backward recursion, and the log-likelihood objective, which is a higher conceptual overhead than BPE.

The practical differences between the two approaches are significant in several dimensions. BPE vocabularies tend to contain more "greedy" tokens: subwords that are very common in their raw character form but may not align well with linguistic units. For example, BPE might create the token "##ing" (a suffix) early in its merge sequence simply because the character trigram "ing" is common, even if this particular three-character token is redundant given tokens like "##running," "##playing," and "##walking" that appear frequently enough to be learned later. The unigram LM is more principled: every token that survives pruning increases the corpus likelihood, meaning it is needed to compress the data more efficiently than the remaining vocabulary could without it.

Empirically, both approaches produce similar-quality tokenizations, but the unigram LM's probabilistic foundation enables subword regularization, which provides a meaningful edge in training downstream models. On tasks where morphological variation is common, such as machine translation with inflected languages or document classification across formal and informal registers, models trained with unigram sampling tend to generalize better.

One important computational tradeoff is that unigram LM training involves running EM and evaluating all segmentations, which is more expensive than BPE's simple frequency-counting approach. For large corpora with hundreds of gigabytes of text, this cost can be substantial. In practice, tokenizer training on large corpora is done once and the result is reused, so the higher training cost is typically acceptable. The cost is paid during tokenizer training, not during downstream model training or inference.

WordPiece, the third major subword algorithm, falls between BPE and unigram LM. Like BPE, it is constructive (merge-based), but it selects merges based on a likelihood criterion rather than raw frequency. WordPiece does not naturally support sampling, but its likelihood-based training gives it some of the principled quality of the unigram LM approach without the full EM machinery.