Unigram Language Model Tokenization › The Core Idea: Tokenization as Probabilistic Modeling

To understand why the unigram approach is principled, you need to appreciate just how ambiguous tokenization really is. Given the word "unhappiness," is the right split ["un", "happiness"], ["un", "happy", "ness"], ["unhappy", "ness"], or ["u", "n", "happiness"]? All of these are valid subword segmentations. Each captures different structural information about the word. The first recognizes the prefix "un" and treats "happiness" as a semantic unit. The second decomposes the word into three meaningful morphemes. The third groups the adjectival stem with its negative prefix and separates the nominalization suffix. The fourth is arguably silly but still technically valid as a string decomposition.

BPE picks one segmentation deterministically based on which merges were applied in order. This determinism feels clean, but it hides a real ambiguity. The "best" segmentation according to BPE is best only in the sense that it follows the greedy merge sequence BPE happened to learn. Another training run on a slightly different corpus might produce a different merge sequence and therefore a different segmentation of the same word. The choice of segmentation is not linguistic ground truth; it is an artifact of the algorithm.

The unigram LM takes a more honest stance: all segmentations are possible, and we should assign probabilities to each one. The segmentation with the highest probability under our learned model is the best guess, but it is still just a guess. Other segmentations are not wrong; they are just less likely given the training data. This probabilistic framing immediately permits sampling: we can draw segmentations from the distribution instead of always taking the argmax, and that diversity turns out to be tremendously useful during model training.

The unigram language model treats each token in a segmentation as independently drawn from a probability distribution over the vocabulary. The word "unigram" in the algorithm's name refers to this independence assumption: we model each token with a unigram (single-token) probability, ignoring context. Given a vocabulary $V$ where each token $x_i \in V$ has an assigned probability $P(x_i)$, the probability of a particular segmentation $\mathbf{x} = (x_1, x_2, \ldots, x_m)$ of a word is computed by multiplying together the individual token probabilities:

$$P(\mathbf{x}) = \prod_{i=1}^{m} P(x_i)$$

where:

• $\mathbf{x}$: a specific segmentation, meaning a specific sequence of subword tokens that together reconstruct the original word
• $m$: the number of tokens in this segmentation
• $P(x_i)$: the probability of the $i$-th token in the segmentation, estimated from training data
• $\prod$: the product over all token probabilities, which reflects the independence assumption that each token contributes multiplicatively to the overall probability

Why does multiplication make sense here? Because we are computing the joint probability of a sequence of independent events. If flipping heads has probability 0.5, the probability of flipping heads three times in a row is $0.5 \times 0.5 \times 0.5 = 0.125$. Analogously, if seeing token "un" has probability 0.05 and seeing "happy" has probability 0.04, the probability of the segmentation ["un", "happy"] under the independence assumption is $0.05 \times 0.04 = 0.002$.

This "unigram" formulation assumes tokens are independent. In reality, adjacent tokens are not independent: "un" appearing next to "happy" is far more likely than "un" appearing next to "ness." But the independence simplification makes the math tractable and the algorithm practical. More sophisticated models that capture token-to-token dependencies would require exponentially more parameters and far more complex inference algorithms. The unigram approximation gives us something that is good enough to rank segmentations effectively while remaining computationally manageable.

The best segmentation is the one that maximizes this probability. Among all ways to split "unhappiness," we choose the segmentation whose constituent tokens have the highest combined probability. Intuitively, the best split uses common, well-established subwords rather than obscure fragments. If "happiness" is a common enough subword that its probability exceeds the combined probability of "happy" and "ness," then ["un", "happiness"] beats ["un", "happy", "ness"]. The model's learned probabilities encode collective evidence from the training corpus about which subword units are truly productive.

this probabilistic view converts tokenization from a rule-based string operation into an inference problem. You are asking: given the evidence encoded in training data token frequencies, what is the most probable explanation for how this word was constructed from subword units? This reframing connects tokenization to a long tradition of probabilistic modeling in NLP and makes the algorithm amenable to the toolkit of statistical inference.

Before diving into the training algorithm, it is worth pausing to understand why the subword level is the right granularity for this probabilistic model. Word-level probabilities fail for rare and out-of-vocabulary words: a model that assigns probability only to whole words seen in training cannot handle new words at all. Character-level probabilities solve the coverage problem but produce extremely long sequences where each character must be processed independently, which is both slow and fails to encode the morphological structure that makes words predictable.

The subword level hits the sweet spot. Subwords are short enough that a large fraction of them appear frequently in training data, giving reliable probability estimates. They are long enough to encode meaningful linguistic units like prefixes, suffixes, and stems. And the vocabulary is small enough to be manageable as a discrete probability distribution. The unigram LM formalizes this intuition by learning exactly which subwords deserve their own probability mass and which are better decomposed into smaller pieces.