Unigram Language Model Tokenization › Worked Example: Segmenting a Word

Let's trace through the core ideas with a small, concrete example. Working through a numerical example makes the abstract formulas tangible and reveals how the algorithm's decisions arise from the learned probabilities. Suppose our vocabulary contains these tokens with their probabilities:

These probabilities reflect a hypothetical trained model. "un" is the most probable token (0.05). This reflects its frequent use as a prefix across many words. "happiness" has the lowest probability (0.015) because it is a longer, more specific token that appears less frequently in a typical corpus.

To segment "unhappiness," we consider the main candidate segmentations and compute the probability of each:

Segmentation A: ["un", "happy", "ness"]

We compute the probability as the product of the three token probabilities:

$$P_A = P(\text{un}) \times P(\text{happy}) \times P(\text{ness}) = 0.05 \times 0.04 \times 0.03 = 0.000060$$

In log space: $\log_{10} P_A = -1.301 + (-1.398) + (-1.523) = -4.222$

Segmentation B: ["unhappy", "ness"]

$$P_B = P(\text{unhappy}) \times P(\text{ness}) = 0.02 \times 0.03 = 0.000600$$

In log space: $\log_{10} P_B = -1.699 + (-1.523) = -3.222$

Segmentation C: ["un", "happiness"]

$$P_C = P(\text{un}) \times P(\text{happiness}) = 0.05 \times 0.015 = 0.000750$$

In log space: $\log_{10} P_C = -1.301 + (-1.824) = -3.125$

Comparing these values: $P_C = 0.000750 > P_B = 0.000600 > P_A = 0.000060$. The Viterbi algorithm chooses segmentation C, ["un", "happiness"], as the most probable.

This result deserves careful interpretation. Notice how the model preferred the two-token split using the compound token "happiness" over the three-token split ["un", "happy", "ness"]. Even though "happiness" has a lower individual probability (0.015) than "ness" (0.03) or "happy" (0.04), the two-token segmentation wins because multiplying two moderately small numbers gives a larger result than multiplying three small numbers. Each additional token in a segmentation multiplies in another factor less than one, so shorter segmentations tend to win unless the longer segmentation's tokens are individually much more probable.

This is the unigram LM's built-in Occam's razor: all else being equal, simpler (fewer tokens) segmentations are favored. The model will only use more tokens when the increased specificity of a longer segmentation brings enough probability gain to overcome the cost of the extra multiplication.

Now let's trace through the Viterbi recurrence for "unhappiness" (of length 11). We build the best table position by position:

• $\text{best}[0] = 0$ (empty string)
• $\text{best}[2] = \text{best}[0] + \log P(\text{un}) = 0 + (-1.301) = -1.301$ (token "un" from positions 0-2)
• At position 9 ("unhappines"), we find "happiness" runs from positions 2-11, but we are only at position 9 here, so we must continue.
• $\text{best}[11]$: we check token "happiness" from positions 2-11. $\text{best}[2] + \log P(\text{happiness}) = -1.301 + (-1.824) = -3.125$. We also check "ness" from positions 7-11: $\text{best}[7] + \log P(\text{ness})$, where $\text{best}[7]$ was computed as $\text{best}[0] + \log P(\text{unhappy}) = -1.699$ (if "unhappy" runs from 0 to 7). So this path gives $-1.699 + (-1.523) = -3.222$.

The maximum at position 11 is $-3.125$ (from the path via "un" then "happiness"), confirming our earlier calculation. The backtracking pointer at position 11 points to position 2, and the pointer at position 2 points to position 0, recovering the segmentation ["un", "happiness"].