WordPiece › A Worked Example: Tracing the Training Algorithm

Let us work through a minimal numerical example to make the training algorithm concrete. We will use a tiny corpus and small vocabulary to trace exactly how WordPiece scores differ from BPE scores.

Suppose our training corpus contains four words with the following frequencies:

Step 1: Initialization. We split each word into characters with ## prefixes:

Step 2: Compute token counts. Adding up occurrences across the corpus (multiplied by word frequencies):

• l: 5 + 2 = 7 (from "low" and "lower")
• ##o: 5 + 2 = 7
• ##w: 5 + 2 + 6 = 13 (from "low", "lower", "newest")
• ##e: 2 + 6 + 6 + 3 = 17 (from "lower": 1, "newest": 2, "wider": 1)
• ##r: 2 + 3 = 5
• n: 6
• ##s: 6
• ##t: 6
• w: 3 (from "wider")
• ##i: 3
• ##d: 3

Total token count: 7 + 7 + 13 + 17 + 5 + 6 + 6 + 6 + 3 + 3 + 3 = 76

Step 3: Score the candidate pairs. For the pair (##e, ##r), which appears in "lower" (count 2) and "wider" (count 3), making 5 occurrences:

$$P(\text{\#\#e}) = 17/76 \approx 0.224, \quad P(\text{\#\#r}) = 5/76 \approx 0.066$$ $$P(\text{\#\#e, \#\#r}) = 5/76 \approx 0.066$$ $$\text{score}(\text{\#\#e}, \text{\#\#r}) = \frac{0.066}{0.224 \times 0.066} \approx \frac{0.066}{0.0148} \approx 4.46$$

Now compare this to the pair (##e, ##w) in "newest" (count 6 but ##e appears twice in "newest", specifically the first ##e precedes ##w):

$$P(\text{\#\#e, \#\#w}) = 6/76 \approx 0.079$$ $$\text{score}(\text{\#\#e}, \text{\#\#w}) = \frac{0.079}{0.224 \times 0.171} \approx \frac{0.079}{0.038} \approx 2.08$$

where $P(\text{\#\#w}) = 13/76 \approx 0.171$.

Even though ##e followed by ##w appears 6 times (more often than ##e followed by ##r), the WordPiece score favors ##er because ##r is much less common than ##w. When ##e and ##r appear together, it is more informative: the relatively rare ##r is almost exclusively found after ##e. A BPE tokenizer would have chosen (##e, ##w) at this step because 6 > 5. WordPiece chooses (##e, ##r) because the PMI score is higher.

Step 4: After the merge. We add ##er to the vocabulary. The words "lower" and "wider" now have splits l ##o ##w ##er and w ##i ##d ##er. We recompute all scores and continue until we reach our target vocabulary size.

This example illustrates the core intuition: WordPiece prefers merges where the pair is mutually informative, not just frequently co-occurring in absolute terms.