WordPiece › From Frequency to Likelihood: The Core Difference

The BPE algorithm, as you learned in the previous chapter, operates on a simple principle: count every adjacent pair of symbols in the current vocabulary, find the most frequent pair, and merge it. Repeat until you reach the target vocabulary size. This works well in practice, but it has a blind spot. Frequency counts treat all occurrences equally. A pair that appears 1,000 times is ranked above a pair that appears 999 times regardless of whether those occurrences are statistically meaningful.

WordPiece formalizes a different intuition. If two subwords $A$ and $B$ appear together as $AB$ much more often than you would expect if they combined randomly, they likely form a meaningful unit. The merge is worthwhile because $AB$ is frequent and because the co-occurrence is informative.

To understand why this distinction matters, consider an analogy. Suppose you are studying which words in a library catalog tend to appear together in book titles. The words "the" and "of" are individually very common, and consequently they co-occur frequently in titles like "The History of Rome" or "The Chronicles of Narnia." But their co-occurrence is almost entirely explained by the fact that both are common function words. They are not specifically attracted to each other; they are just both everywhere. Now compare this to the words "neural" and "##network": these co-occur nearly every time "neural" appears because "network" almost always follows "neural" in technical contexts. The co-occurrence is informative about the structure of the domain. WordPiece's scoring is designed to reward the second type of pair and discount the first.

This difference plays out across the vocabulary in subtle but important ways. BPE will sometimes create tokens that are frequent but semantically empty, because high-frequency function word fragments happen to appear together often. WordPiece tends to avoid this by normalizing the raw co-occurrence by each token's individual popularity. The result is a vocabulary that more reliably corresponds to linguistically meaningful units.

WordPiece frames tokenizer training as a language modeling problem. Given the current vocabulary $V$, we can assign a probability to any word by treating it as a sequence of subword tokens and multiplying their individual probabilities. For a word $w$ tokenized as $t_1 t_2 \cdots t_k$, the probability under a unigram language model is:

$$P(w) = \prod_{i=1}^{k} P(t_i)$$

where each token probability $P(t_i)$ is estimated from corpus frequency:

$$P(t) = \frac{\text{count}(t)}{\sum_{t' \in V} \text{count}(t')}$$

where:

• $P(w)$: the probability assigned to word $w$ under the current tokenization
• $t_1, t_2, \ldots, t_k$: the subword tokens that word $w$ is split into under the current vocabulary
• $P(t_i)$: the unigram probability of token $t_i$, estimated from how often it appears across the corpus
• $\text{count}(t)$: the number of times token $t$ appears in the training corpus after tokenizing every word with the current vocabulary
• $\sum_{t' \in V} \text{count}(t')$: the total count of all tokens, which normalizes the distribution to sum to one

This formulation is called a unigram language model because it treats each token as independently distributed: the probability of a token does not depend on what came before it. In practice, language clearly is not unigram, but this simplification is a useful approximation for the purpose of scoring which vocabulary gives the most efficient representation of the corpus.

The overall training objective is to maximize the log-likelihood of the entire corpus:

$$\mathcal{L} = \sum_{w \in \text{corpus}} \text{count}(w) \cdot \log P(w)$$

where:

• $\text{count}(w)$: the number of times word $w$ appears in the training corpus
• $\log P(w)$: the log probability assigned to word $w$ under the current vocabulary and tokenization

When we consider merging two tokens $A$ and $B$ into a new token $AB$, we ask: by how much does this merge increase $\mathcal{L}$? The candidate merge is the one that maximizes this gain.

Why does this make sense? Notice that maximizing log-likelihood is equivalent to finding the vocabulary that assigns the highest probability to the training corpus. A vocabulary that uses many short, individually common tokens will assign each word a product of many small probabilities, giving a low overall probability. A vocabulary that uses longer, more specific tokens can assign fewer but higher-probability factors to the same words, giving a higher overall probability. The likelihood criterion is therefore pushing the vocabulary toward tokens that efficiently explain the data: not too short (needing many multiplications of small probabilities) and not too rare (needing very small probabilities for the individual tokens).

The gain from merging $A$ and $B$ can be derived from the change in log-likelihood. Before the merge, every occurrence of $AB$ in the corpus contributes $\log P(A) + \log P(B)$ to the log-likelihood. After the merge, those same occurrences contribute $\log P(AB)$. The change in log-likelihood per co-occurrence is:

$$\Delta\mathcal{L}(A, B) = \log P(AB) - \log P(A) - \log P(B)$$

To see where this comes from, observe that when we add $AB$ to the vocabulary, every position in the corpus where $A$ was immediately followed by $B$ now becomes a single token $AB$. The log-likelihood changes exactly by the amount shown above, multiplied by the number of positions where $AB$ appears. We can factor that count into the probabilities:

$$\begin{aligned} \Delta\mathcal{L}_{\text{total}} &= \text{count}(AB) \cdot [\log P(AB) - \log P(A) - \log P(B)] \\ &= \text{count}(AB) \cdot \log \frac{P(AB)}{P(A) \cdot P(B)} \end{aligned}$$

Since $\text{count}(AB)$ is always positive, the sign of the total gain is determined entirely by the ratio $P(AB) / (P(A) \cdot P(B))$. Exponentiating the per-occurrence change gives the score used in practice:

$$\text{score}(A, B) = \frac{P(AB)}{P(A) \cdot P(B)}$$

where:

• $P(AB)$: the probability of the merged token $AB$, estimated from how often $AB$ appears as a unit after the merge
• $P(A)$: the probability of token $A$ under the current vocabulary
• $P(B)$: the probability of token $B$ under the current vocabulary

This score is a form of pointwise mutual information (PMI). A score greater than 1 means $A$ and $B$ appear together more often than chance. A score of exactly 1 would mean the co-occurrence is entirely explained by their individual frequencies. WordPiece selects the candidate merge with the highest score at each step.

Why does this formula make sense? Notice that $P(A) \cdot P(B)$ represents the probability of seeing $A$ and $B$ together if they were completely independent: the joint probability under a no-interaction assumption. The numerator $P(AB)$ represents the actual observed probability of the pair. When the ratio exceeds 1, the pair co-occurs more than independence would predict, indicating a statistical dependency. The higher the ratio, the more informative the merge: the two tokens are more strongly "attracted" to each other than their individual frequencies would suggest.

Consider a concrete case. Suppose the word "playing" appears 10,000 times in the corpus, and the pair ("play", "##ing") has a high frequency because "playing" is so common. BPE would merge this pair. But WordPiece's score normalizes by $P(\text{play}) \cdot P(\text{\#\#ing})$. If "play" and "##ing" are both individually very common (appearing in many other words), their individual probabilities are large, and the score $P(\text{play} \cdot \text{\#\#ing}) / (P(\text{play}) \cdot P(\text{\#\#ing}))$ may not be especially high. The individual popularity of each token dilutes the informativeness of their co-occurrence.

Contrast this with a less frequent but more strongly associated pair, such as the components of a technical suffix that only occurs together. Even if this pair appears less often in absolute terms, if its co-occurrence is far above what the individual token frequencies would predict, WordPiece will rank it higher.

In practice, WordPiece vocabularies tend to preserve linguistically meaningful units like common prefixes, suffixes, and root forms more reliably than BPE vocabularies of the same size, because PMI naturally rewards co-occurrences that carry information rather than co-occurrences that are simply frequent by accident. This is not a dramatic difference: both BPE and WordPiece produce subword vocabularies that work well in practice. But the probabilistic foundation gives WordPiece a principled justification and tends to produce slightly cleaner vocabulary boundaries on morphologically rich text.

WordPiece is more expensive to compute than BPE. At each training step, BPE simply counts pair frequencies, which is $O(N)$ in the corpus size. WordPiece must compute token probabilities across the entire corpus, which requires tracking all token counts and normalizing. Modern implementations use efficient data structures to keep this tractable, but the fundamental computation is more involved. This is partly why BPE has been preferred in some contexts where training speed matters.