WordPiece: BERT's Subword Tokenization Algorithm Explained

Subword tokenization took a decisive turn in 2018 when Google published BERT. While Byte Pair Encoding had already shown that splitting words into pieces rather than treating them as atomic units could dramatically reduce vocabulary problems, BERT's authors chose a different algorithm for their tokenizer: WordPiece. On the surface, WordPiece and BPE look similar. Both start from individual characters, both build up a vocabulary by iteratively merging units, and both produce the same kind of subword tokens you recognize in practice: things like play, ##ing, un, ##der, ##stand. But the way they decide which pairs to merge is fundamentally different, and that difference shapes everything from what the vocabulary captures to how the tokenizer behaves on rare words.

As we covered in the BPE chapter, BPE chooses its merges greedily by frequency: whatever pair appears most often in the training corpus gets merged next. WordPiece takes a probabilistic perspective instead. Rather than asking "which pair appears most often?", it asks "which merge increases the likelihood of the training data the most?" These two questions sound similar but lead to vocabularies with different properties. WordPiece tends to prefer merges that capture meaningful morphological units, because those units appear together more often than you would expect from their individual frequencies alone.

Think of the difference this way: BPE is like a hiring manager who promotes whoever has worked the most hours, while WordPiece is like one who promotes whoever adds the most value beyond what could be explained by their general role. The BPE criterion is simple and transparent; the WordPiece criterion is more selective and rewards statistical association between token pairs. A pair of tokens that are both extremely common individually might co-occur often simply because each one appears everywhere, but WordPiece's scoring would not reward that pairing highly. Only pairs whose co-occurrence is informative, exceeding what their individual popularities would predict, climb to the top of WordPiece's merge queue.

morphologically meaningful units tend to have strong statistical associations. The suffix ##ing appears after play far more often than you would predict from knowing only how often play and ##ing each appear independently. That surplus signal is exactly what WordPiece's scoring captures. BPE would only know that "playing" is frequent. WordPiece additionally knows that "play" and "##ing" are unusually attracted to each other, beyond what their separate frequencies would predict.

This chapter covers how WordPiece's likelihood-based merge criterion works, how the greedy tokenization algorithm applies a learned vocabulary to new text, the ## prefix convention that marks continuation subwords, how BERT uses WordPiece in practice, and how to train a WordPiece tokenizer using the HuggingFace tokenizers library. We will also work through a step-by-step numerical example of the training algorithm, examine the limitations of the approach, and understand why BERT's choice of WordPiece mattered for the field of NLP.

The BPE algorithm, as you learned in the previous chapter, operates on a simple principle: count every adjacent pair of symbols in the current vocabulary, find the most frequent pair, and merge it. Repeat until you reach the target vocabulary size. This works well in practice, but it has a blind spot. Frequency counts treat all occurrences equally. A pair that appears 1,000 times is ranked above a pair that appears 999 times regardless of whether those occurrences are statistically meaningful.

WordPiece formalizes a different intuition. If two subwords $A$ and $B$ appear together as $AB$ much more often than you would expect if they combined randomly, they likely form a meaningful unit. The merge is worthwhile because $AB$ is frequent and because the co-occurrence is informative.

To understand why this distinction matters, consider an analogy. Suppose you are studying which words in a library catalog tend to appear together in book titles. The words "the" and "of" are individually very common, and consequently they co-occur frequently in titles like "The History of Rome" or "The Chronicles of Narnia." But their co-occurrence is almost entirely explained by the fact that both are common function words. They are not specifically attracted to each other; they are just both everywhere. Now compare this to the words "neural" and "##network": these co-occur nearly every time "neural" appears because "network" almost always follows "neural" in technical contexts. The co-occurrence is informative about the structure of the domain. WordPiece's scoring is designed to reward the second type of pair and discount the first.

This difference plays out across the vocabulary in subtle but important ways. BPE will sometimes create tokens that are frequent but semantically empty, because high-frequency function word fragments happen to appear together often. WordPiece tends to avoid this by normalizing the raw co-occurrence by each token's individual popularity. The result is a vocabulary that more reliably corresponds to linguistically meaningful units.

WordPiece frames tokenizer training as a language modeling problem. Given the current vocabulary $V$, we can assign a probability to any word by treating it as a sequence of subword tokens and multiplying their individual probabilities. For a word $w$ tokenized as $t_1 t_2 \cdots t_k$, the probability under a unigram language model is:

where each token probability $P(t_i)$ is estimated from corpus frequency:

where:

• $P(w)$: the probability assigned to word $w$ under the current tokenization
• $t_1, t_2, \ldots, t_k$: the subword tokens that word $w$ is split into under the current vocabulary
• $P(t_i)$: the unigram probability of token $t_i$, estimated from how often it appears across the corpus
• $\text{count}(t)$: the number of times token $t$ appears in the training corpus after tokenizing every word with the current vocabulary
• $\sum_{t' \in V} \text{count}(t')$: the total count of all tokens, which normalizes the distribution to sum to one

This formulation is called a unigram language model because it treats each token as independently distributed: the probability of a token does not depend on what came before it. In practice, language clearly is not unigram, but this simplification is a useful approximation for the purpose of scoring which vocabulary gives the most efficient representation of the corpus.

The overall training objective is to maximize the log-likelihood of the entire corpus:

where:

• $\text{count}(w)$: the number of times word $w$ appears in the training corpus
• $\log P(w)$: the log probability assigned to word $w$ under the current vocabulary and tokenization

When we consider merging two tokens $A$ and $B$ into a new token $AB$, we ask: by how much does this merge increase $\mathcal{L}$? The candidate merge is the one that maximizes this gain.

Why does this make sense? Notice that maximizing log-likelihood is equivalent to finding the vocabulary that assigns the highest probability to the training corpus. A vocabulary that uses many short, individually common tokens will assign each word a product of many small probabilities, giving a low overall probability. A vocabulary that uses longer, more specific tokens can assign fewer but higher-probability factors to the same words, giving a higher overall probability. The likelihood criterion is therefore pushing the vocabulary toward tokens that efficiently explain the data: not too short (needing many multiplications of small probabilities) and not too rare (needing very small probabilities for the individual tokens).

The gain from merging $A$ and $B$ can be derived from the change in log-likelihood. Before the merge, every occurrence of $AB$ in the corpus contributes $\log P(A) + \log P(B)$ to the log-likelihood. After the merge, those same occurrences contribute $\log P(AB)$. The change in log-likelihood per co-occurrence is:

To see where this comes from, observe that when we add $AB$ to the vocabulary, every position in the corpus where $A$ was immediately followed by $B$ now becomes a single token $AB$. The log-likelihood changes exactly by the amount shown above, multiplied by the number of positions where $AB$ appears. We can factor that count into the probabilities:

Since $\text{count}(AB)$ is always positive, the sign of the total gain is determined entirely by the ratio $P(AB) / (P(A) \cdot P(B))$. Exponentiating the per-occurrence change gives the score used in practice:

where:

• $P(AB)$: the probability of the merged token $AB$, estimated from how often $AB$ appears as a unit after the merge
• $P(A)$: the probability of token $A$ under the current vocabulary
• $P(B)$: the probability of token $B$ under the current vocabulary

This score is a form of pointwise mutual information (PMI). A score greater than 1 means $A$ and $B$ appear together more often than chance. A score of exactly 1 would mean the co-occurrence is entirely explained by their individual frequencies. WordPiece selects the candidate merge with the highest score at each step.

Why does this formula make sense? Notice that $P(A) \cdot P(B)$ represents the probability of seeing $A$ and $B$ together if they were completely independent: the joint probability under a no-interaction assumption. The numerator $P(AB)$ represents the actual observed probability of the pair. When the ratio exceeds 1, the pair co-occurs more than independence would predict, indicating a statistical dependency. The higher the ratio, the more informative the merge: the two tokens are more strongly "attracted" to each other than their individual frequencies would suggest.

Consider a concrete case. Suppose the word "playing" appears 10,000 times in the corpus, and the pair ("play", "##ing") has a high frequency because "playing" is so common. BPE would merge this pair. But WordPiece's score normalizes by $P(\text{play}) \cdot P(\text{\#\#ing})$. If "play" and "##ing" are both individually very common (appearing in many other words), their individual probabilities are large, and the score $P(\text{play} \cdot \text{\#\#ing}) / (P(\text{play}) \cdot P(\text{\#\#ing}))$ may not be especially high. The individual popularity of each token dilutes the informativeness of their co-occurrence.

Contrast this with a less frequent but more strongly associated pair, such as the components of a technical suffix that only occurs together. Even if this pair appears less often in absolute terms, if its co-occurrence is far above what the individual token frequencies would predict, WordPiece will rank it higher.

In practice, WordPiece vocabularies tend to preserve linguistically meaningful units like common prefixes, suffixes, and root forms more reliably than BPE vocabularies of the same size, because PMI naturally rewards co-occurrences that carry information rather than co-occurrences that are simply frequent by accident. This is not a dramatic difference: both BPE and WordPiece produce subword vocabularies that work well in practice. But the probabilistic foundation gives WordPiece a principled justification and tends to produce slightly cleaner vocabulary boundaries on morphologically rich text.

WordPiece is more expensive to compute than BPE. At each training step, BPE simply counts pair frequencies, which is $O(N)$ in the corpus size. WordPiece must compute token probabilities across the entire corpus, which requires tracking all token counts and normalizing. Modern implementations use efficient data structures to keep this tractable, but the fundamental computation is more involved. This is partly why BPE has been preferred in some contexts where training speed matters.

WordPiece uses a simple but important convention to distinguish between tokens that start a word and tokens that appear in the middle or end of a word. Tokens that can begin a word carry no special marker. Tokens that are continuation pieces (meaning they follow some preceding token within the same word) are prefixed with ##.

This matters because "ing" and "##ing" are different tokens in the vocabulary. The token "ing" might appear as an independent word in some context (rare in English, but the point stands). The token "##ing" unambiguously means "a suffix that attaches to the previous token as part of the same word." Without this distinction, a model would have no way to tell whether the token "ing" was a standalone word or the suffix of a longer word. The ## prefix bakes word boundary information directly into the token identity.

Think of the ## prefix as a grammatical marker analogous to a hyphen in a dictionary entry. When a dictionary entry shows "re-" as a prefix or "-ing" as a suffix, it is communicating that these pieces attach to other words, not that they stand alone. The ## convention does the same thing for subword tokens: it signals attachment to the preceding piece of the same word.

When WordPiece tokenizes the word "playing", it might produce:

When it tokenizes "playing a game", the full output might be:

The ## prefix tells any downstream model (like BERT) that "##ing" is part of the same word as "play", not an independent token. BERT uses this information when determining word boundaries and when performing tasks like named entity recognition that require knowing which tokens belong to the same word. For example, when BERT is used for sequence labeling tasks like NER, the model typically assigns a label to the first token of each word and ignores or copies the label for ## continuation tokens, since a multi-token word like "un##play##ed" should receive a single entity label rather than one per subword.

The convention also matters during decoding. To reconstruct the original text from WordPiece tokens, you concatenate tokens that start with ## directly to the previous token, and insert a space before tokens that do not start with ##. So ["play", "##ing", "a", "game"] reconstructs as "playing a game".

One limitation of the ## convention appears in languages without clear word boundaries. In Japanese, for instance, text is not separated by spaces, so the notion of "word-initial" versus "word-internal" is less clear. In these settings, pre-tokenization strategies that impose pseudo-word boundaries are needed before WordPiece can operate effectively. This is one reason why multilingual models and language-specific models sometimes use different tokenization approaches that do not rely on the ## convention.

WordPiece training begins with a character-level vocabulary and builds up from there. The procedure has three stages: initialization, the merge loop, and final vocabulary construction. The overall structure is nearly identical to BPE; the critical difference is only in how merges are scored.

Understanding the training algorithm matters because it explains both what the vocabulary contains and why it looks the way it does. The tokens in a WordPiece vocabulary are not arbitrary: each one was selected because, at the moment it was created, it represented the most statistically informative unit to add, given everything that had been added before. The history of merges shapes the final vocabulary in ways that are not always obvious from looking at the tokens alone.

Initialization. The corpus is segmented into individual characters. Every word is split into its constituent characters, with the first character left as-is and every subsequent character prefixed with ##. For example, "unbelievable" becomes:

The initial vocabulary consists of all unique characters that appear in the corpus, plus a few special tokens (typically [UNK], [CLS], [SEP], [PAD], [MASK] for BERT). This initialization guarantees that every possible input can be tokenized: in the worst case, each character becomes its own token.

The ## prefix on continuation characters is applied from the very beginning, not added later. This means the character e and the continuation character ##e are different entries in the initial vocabulary, and they remain different throughout training. A merge that joins e with something after it will create a word-initial sequence, while a merge that joins ##e with something after it will create a word-internal sequence. The distinction is preserved at every level of the vocabulary.

Merge loop. At each iteration, the algorithm computes the score $P(AB) / (P(A) \cdot P(B))$ for every adjacent pair of tokens that appears in the corpus. It selects the pair with the highest score and merges it into a new token, updating all affected words in the corpus. This process repeats until the vocabulary reaches the target size.

After each merge, all token probabilities must be recomputed, because the merge changes which tokens exist and how often each appears. This is what makes WordPiece training more expensive than BPE: the full probability recalculation happens at each step. In practice, efficient implementations batch these updates and avoid recomputing from scratch by maintaining running counts that are updated locally around the affected pairs.

Final vocabulary. The result is a vocabulary that contains whole words (those frequent enough to be merged all the way), common subword pieces (frequent morphological components), and individual characters as a fallback. In BERT's case, after training on BookCorpus and Wikipedia, the vocabulary of 30,000 merges produces roughly 30,522 tokens including the special tokens.

The key difference from BPE at this algorithmic level is only in the scoring: BPE counts raw co-occurrences; WordPiece computes the PMI-based score. The overall loop structure is nearly identical. This means that if you already understand BPE's training loop, you understand WordPiece: you just swap out one scoring formula for another, and the rest follows.

Let us work through a minimal numerical example to make the training algorithm concrete. We will use a tiny corpus and small vocabulary to trace exactly how WordPiece scores differ from BPE scores.

Suppose our training corpus contains four words with the following frequencies:

Step 1: Initialization. We split each word into characters with ## prefixes:

Step 2: Compute token counts. Adding up occurrences across the corpus (multiplied by word frequencies):

• l: 5 + 2 = 7 (from "low" and "lower")
• ##o: 5 + 2 = 7
• ##w: 5 + 2 + 6 = 13 (from "low", "lower", "newest")
• ##e: 2 + 6 + 6 + 3 = 17 (from "lower": 1, "newest": 2, "wider": 1)
• ##r: 2 + 3 = 5
• n: 6
• ##s: 6
• ##t: 6
• w: 3 (from "wider")
• ##i: 3
• ##d: 3

Total token count: 7 + 7 + 13 + 17 + 5 + 6 + 6 + 6 + 3 + 3 + 3 = 76

Step 3: Score the candidate pairs. For the pair (##e, ##r), which appears in "lower" (count 2) and "wider" (count 3), making 5 occurrences:

Now compare this to the pair (##e, ##w) in "newest" (count 6 but ##e appears twice in "newest", specifically the first ##e precedes ##w):

where $P(\text{\#\#w}) = 13/76 \approx 0.171$.

Even though ##e followed by ##w appears 6 times (more often than ##e followed by ##r), the WordPiece score favors ##er because ##r is much less common than ##w. When ##e and ##r appear together, it is more informative: the relatively rare ##r is almost exclusively found after ##e. A BPE tokenizer would have chosen (##e, ##w) at this step because 6 > 5. WordPiece chooses (##e, ##r) because the PMI score is higher.

Step 4: After the merge. We add ##er to the vocabulary. The words "lower" and "wider" now have splits l ##o ##w ##er and w ##i ##d ##er. We recompute all scores and continue until we reach our target vocabulary size.

This example illustrates the core intuition: WordPiece prefers merges where the pair is mutually informative, not just frequently co-occurring in absolute terms.

Once training is complete, tokenizing new text with WordPiece uses a different strategy than BPE. BPE applies its learned merge rules sequentially in the order they were learned. WordPiece uses a greedy longest-match-first approach instead.

For each word in the input, the algorithm attempts to find the longest prefix of the word that exists in the vocabulary. It takes that prefix as the first token, then repeats for the remaining suffix. The suffix tokens receive the ## prefix. If at any point no prefix at all exists in the vocabulary (not even the single-character fallback), the entire word becomes [UNK].

Think of this as a greedy left-to-right scan through the word, always grabbing the biggest bite the vocabulary allows at each step. The algorithm never backtracks: once it has committed to a prefix, it continues from that point. This greedy nature means it can occasionally produce suboptimal segmentations for ambiguous cases, but those cases are rare in practice because the vocabulary is designed to give good coverage.

The procedure for tokenizing a single word is:

1. Start at the beginning of the word
2. Find the longest prefix that is in the vocabulary
3. Emit that prefix as a token
4. If there is remaining text, prepend ## to it and go back to step 2
5. If at any point no prefix (not even the first character) is in the vocabulary, replace the entire word with [UNK]

This greedy algorithm is fast and deterministic: it always produces the same segmentation for a given word and vocabulary. Unlike BPE, which requires storing and re-applying ordered merge rules, WordPiece inference requires only a lookup in the vocabulary dictionary, which makes it slightly simpler to implement and slightly less sensitive to the order of training merges. There is no list of merge rules to store: the vocabulary itself encodes all the information needed to tokenize, because the greedy algorithm implicitly reconstructs the appropriate segmentation.

This difference in inference approach is worth emphasizing. BPE tokenization is inherently rule-based: you apply rule 1, then rule 2, then rule 3, in the order they were created. The order matters because later rules were designed assuming earlier ones have already been applied. WordPiece inference has no such ordering dependency: you simply look up the longest matching prefix at each position. The vocabulary was built by learning which merges were informative, but the tokenization itself just does dictionary lookups. This makes WordPiece tokenizers easier to serialize, share, and apply in different computational environments.

Let us trace through tokenizing the word "unplayed" using a small WordPiece vocabulary that contains: un, ##play, ##ed, play, ##ing, a, and individual characters including u, ##n, ##p, ##l, ##a, ##y, ##e, ##d.

Starting at the beginning of "unplayed":

1. Try "unplayed" as a whole: not in vocabulary.
2. Try "unplaye": not in vocabulary.
3. Try "unplay": not in vocabulary.
4. Try "unpla": not in vocabulary.
5. Try "unpl": not in vocabulary.
6. Try "unp": not in vocabulary.
7. Try "un": found in vocabulary. Emit un. Remaining: "played".
8. Prepend ## to remaining: "##played".
9. Try "##played": not in vocabulary.
10. Try "##playe": not in vocabulary.
11. Try "##play": found in vocabulary. Emit ##play. Remaining: "ed".
12. Prepend ## to remaining: "##ed".
13. Try "##ed": found in vocabulary. Emit ##ed.

Final tokenization: un, ##play, ##ed.

This greedy search always finds the longest matching prefix at each step. If a character has no match at all (not even as a single character in the vocabulary), the entire word becomes [UNK]. In practice, BERT's vocabulary includes all Unicode characters it encountered during training, so single-character fallbacks are almost always available.

Notice how the algorithm naturally decomposes the word into morphologically meaningful parts: the prefix un-, the root play, and the past tense suffix -ed. This is not coincidental. The WordPiece training procedure, by using PMI-based scoring, tends to merge characters into units that are linguistically meaningful, because such units have strong statistical dependencies. The greedy tokenization then recovers these units efficiently.

Let us trace a second example to see the [UNK] fallback in action. Suppose our vocabulary does not contain the character ##z (because the training corpus contained no words with z after the first character position). If we try to tokenize "puzzling", we would proceed:

1. Try "puzzling": not in vocabulary.
2. Try longer and shorter prefixes until we reach "p": found in vocabulary. Emit p. Remaining: "uzzling".
3. Prepend ##: "##uzzling".
4. No prefix starting with ##u exists (suppose ##u is not in the vocabulary either).
5. The algorithm returns [UNK] for the entire word.

This shows that the [UNK] fallback is triggered by the inability to make any forward progress, not just by failing to match the whole word. The entire word is replaced with [UNK] as a conservative choice. For BERT's actual 30,000-token vocabulary, this almost never happens because the vocabulary covers a wide range of characters.

BERT uses a WordPiece vocabulary of approximately 30,000 tokens. The vocabulary was trained on a large English corpus (BookCorpus plus English Wikipedia). Because the training data was English-only, the vocabulary is optimized for English morphology, though BERT models in other languages and multilingual BERT use vocabularies trained on corresponding corpora.

The choice of vocabulary size was a deliberate engineering decision. At 30,000 tokens, the embedding matrix (which maps each token to a dense vector) contains 30,000 rows. For BERT-base with 768-dimensional embeddings, this is about 23 million parameters just for the embedding table, which is roughly 25% of BERT-base's total parameter count. Going much larger would make the embedding table dominate the model. Going much smaller would produce more out-of-vocabulary tokens and longer sequences. The 30,000-token vocabulary represents a practical balance between coverage and compactness.

The BERT tokenizer adds several special tokens that have fixed positions and meanings. These tokens are not just implementation details: they carry semantic meaning within BERT's pre-training framework.

• [CLS]: Added at the beginning of every input sequence. The BERT model uses the representation of this token as a summary of the entire sequence for classification tasks. During pre-training, BERT learns to encode sequence-level information into the [CLS] position via the next-sentence prediction objective.
• [SEP]: Added at the end of each segment in the input. For tasks that take two sentences (like textual entailment), [SEP] separates the two segments and marks where one ends and the other begins.
• [PAD]: Used to pad shorter sequences to a fixed length in batched processing. BERT uses an attention mask to ignore these padding tokens, so their presence does not affect the model's computations beyond memory layout.
• [MASK]: Used during masked language model pre-training to replace tokens that the model must predict. Approximately 15% of tokens are selected for masking, and of those, 80% are replaced with [MASK], 10% with a random token, and 10% left unchanged.
• [UNK]: The fallback for any word that cannot be tokenized. In practice this is very rare since single characters are always in the vocabulary.

When tokenizing a sentence for BERT, the full process is:

1. Lowercase the text (for the uncased BERT model) or preserve case (for the cased model)
2. Run basic whitespace and punctuation splitting to produce candidate words
3. Apply WordPiece tokenization to each word
4. Prepend [CLS] and append [SEP]
5. Map each token to its integer ID in the vocabulary

The result is a sequence of integer IDs that BERT processes as its input.

BERT's maximum sequence length of 512 tokens is closely tied to the tokenization. Because subword tokenization expands words into multiple tokens, a sentence of 100 words might produce 130-150 tokens after WordPiece splits compound words and rare morphological forms. This expansion factor is typically 1.1 to 1.5 for English text, meaning the 512-token limit corresponds to roughly 300-450 words. For longer documents, practitioners typically truncate or use chunking strategies, which is an active area of research for long-document understanding.

The bert-base-uncased tokenizer lowercases all input before tokenization. This halves the effective vocabulary space for alphabetic characters (no need for both A and a), allowing the vocabulary budget to be spent on more subword units instead. The trade-off is that case information is lost, which hurts performance on tasks where capitalization carries semantic weight (such as named entity recognition, where "Apple" the company and "apple" the fruit are different). The bert-base-cased variant preserves case and uses a separately trained vocabulary; it performs better on such tasks at the cost of a slightly higher unknown token rate.

Let us build this from the ground up. We will implement WordPiece training from scratch to understand the mechanics, then use HuggingFace's tokenizers library to train a production-quality tokenizer.

First, install the necessary packages:

We will implement a minimal WordPiece trainer to show exactly how the merge loop works. This is not optimized for large corpora but is faithful to the algorithm. The implementation has four key functions: one to count word frequencies, one to convert words to their initial character-level representation, one to score all candidate pairs using the PMI formula, and one to perform a single merge step.

The vocabulary reveals the PMI-based selection: continuation tokens that represent common morphological suffixes appear early because they are strongly associated with specific preceding tokens. The ## prefix tokens show how character sequences have been merged into meaningful units.

Let us implement the greedy longest-match-first tokenizer. The key insight is that this tokenizer does not need to know the history of how the vocabulary was built: it only needs the final vocabulary set. Given the vocabulary, it simply scans each word from left to right, grabbing the longest match at each step.

The HuggingFace tokenizers library provides a highly optimized WordPiece implementation. Let us train one on a larger English corpus to see more realistic vocabulary behavior:

The tokenizer splits words like "reconstruction" into subword pieces and marks continuation tokens with ##. Words seen during training are kept whole or split at meaningful morphological boundaries. You can see how the training corpus influences which units become full tokens: words that appeared often in the training data (like "tokenization" and "algorithms") may receive their own vocabulary entries, while rare or unseen words get decomposed into smaller pieces.

In practice, you will almost always use BERT's existing tokenizer rather than training a new one from scratch. The HuggingFace transformers library makes this straightforward:

Notice that BERT's tokenizer automatically wraps the sequence with [CLS] at the start and [SEP] at the end. Long technical words like "parallelization" are split into recognizable morphological pieces. The word "reconstruction" becomes re, ##con, ##struct, ##ion, which corresponds to the prefix "re-", the root component "construct", and the noun suffix "-ion". This decomposition allows BERT to share learned representations across morphologically related forms.

The key parameters for training a WordPiece tokenizer are:

• vocab_size: Controls the final vocabulary size. BERT uses 30,522 for English. Larger vocabularies reduce unknown token rates but require more embedding parameters. The right size depends on the language and the model capacity.
• min_frequency: The minimum number of times a token pair must appear to be eligible for merging. Higher values produce a vocabulary biased toward common patterns and may reject merges for rare but linguistically meaningful units.
• continuing_subword_prefix: The prefix appended to continuation tokens. WordPiece uses ## by convention; changing this would break compatibility with pre-trained BERT models.
• special_tokens: Reserved tokens for task-specific use. BERT's five special tokens ([UNK], [CLS], [SEP], [PAD], [MASK]) occupy fixed positions at the start of the vocabulary.

One practical concern with any subword tokenizer is what happens when it encounters a character that was not present in the training corpus. BPE can produce the [UNK] token for entire unknown words, but in theory a BPE vocabulary trained with single characters as the initial alphabet will always be able to tokenize any text using individual characters as fallback.

WordPiece has the same property: if the vocabulary contains all characters from the training corpus, any new word can always be tokenized character by character, with each character after the first receiving the ## prefix. In the worst case, every character in an unknown word becomes its own token. This character-by-character fallback is not ideal, because individual characters carry very little semantic information on their own. A sequence of character tokens conveys the spelling of a word without providing much help in understanding its meaning. However, it is always preferable to dropping the word entirely, and it allows the model to at least process the input without crashing.

The character fallback has an interesting side effect for sequences from scripts that are completely different from the training language. Suppose BERT's English vocabulary encounters a Greek word. Greek letters are Unicode characters, and if BERT's training corpus included any Greek-script text (even in citations or proper nouns), those characters will be in the vocabulary. Each Greek character becomes its own token. The resulting token sequence conveys the spelling of the Greek word but is extremely long relative to the word length. This makes Greek-script text much more costly in terms of sequence length, which can exhaust BERT's 512-token budget faster than English text.

BERT's vocabulary was trained on English text, so it handles standard ASCII well. It also includes many common characters from other scripts, but encounters difficulty with languages using scripts very different from Latin. This is one reason why multilingual BERT (mBERT) was trained on 104 languages simultaneously, with a larger shared vocabulary of approximately 120,000 tokens to accommodate all those scripts. Even mBERT, however, allocates vocabulary budget unevenly across languages, with high-resource languages getting more tokens and low-resource languages falling back to shorter subword pieces more often. This imbalance is a known weakness of shared multilingual vocabularies.

For truly unseen characters, BERT falls back to [UNK]. This is rare for any language that uses the Latin script but can be frequent for languages using uncommon Unicode ranges that were underrepresented in BERT's training data. Some emoji sequences, mathematical symbols from obscure Unicode planes, and private-use area characters commonly produce [UNK] tokens from standard BERT tokenizers.

Let us visualize two key aspects of WordPiece: how the merge scores differ from raw frequency counts, and how tokenization produces different subword splits across word families.

The word family visualization reveals a key property of WordPiece: morphologically related words share subword tokens. The "play" family all contain the token play, and the "construct" family all share ##struct. This shared subword structure is precisely what allows BERT to generalize across word forms. When the model encounters "deconstructed", it can draw on what it has learned about de, ##con, ##struct, and ##ed independently. Each subword piece carries information learned from all the words in which it appeared, allowing BERT to compose meaning from parts even for words it has never seen as wholes.

WordPiece represents a significant advance over character and word tokenization, but like any engineering solution it makes trade-offs that create real limitations. Understanding these limitations is essential for knowing when to use WordPiece-based systems and when to consider alternatives.

WordPiece's likelihood objective is a strength in theory but comes with practical trade-offs. The PMI-based scoring is computationally more expensive than BPE's raw frequency counting, because each iteration requires computing token probabilities across the entire corpus. Modern implementations are highly optimized, but this cost historically limited WordPiece to settings with significant engineering investment, which is why it remained largely confined to Google's internal systems until BERT made the algorithm public.

The greedy tokenization algorithm, while fast and deterministic, produces only one segmentation of a word. This is unlike the Unigram Language Model tokenizer (which we will cover in the next chapter), which assigns probabilities to all possible segmentations and can sample different splits during training. For many tasks, a single deterministic segmentation is exactly what you want. But for data augmentation or regularization during model training, the lack of stochasticity is a limitation. Research has shown that tokenization-based data augmentation, where you randomly vary how words are segmented, can improve model robustness on morphologically ambiguous inputs. WordPiece's determinism forecloses this option.

WordPiece vocabularies trained on English perform poorly on morphologically rich languages. Languages like Finnish, Turkish, or Hungarian have complex inflectional and derivational systems that can produce thousands of word forms from a single root. A WordPiece vocabulary optimized for English will over-split words in these languages into very short pieces, increasing sequence length and reducing the semantic coherence of individual tokens. For a concrete example: the Finnish word "juoksentelisinkohan" (meaning "I wonder if I should run around") is a single grammatical word that would be split into many short subword pieces by an English WordPiece tokenizer. This length explosion can exhaust the maximum sequence length of transformer models very quickly when processing morphologically rich languages.

The ## prefix convention creates a mild vocabulary inefficiency: every character that can appear as a continuation needs both its plain form (as a word-initial character) and its ## form. For languages like Japanese or Chinese, where there is no notion of whitespace-delimited words, the ## convention becomes largely meaningless. These languages require entirely different pre-tokenization strategies, often treating each character or each Unicode codepoint as a word boundary. SentencePiece's approach of using a special character to mark word beginnings rather than word continuations handles these languages more gracefully.

Another limitation is that the vocabulary is static once trained. If the training corpus is from 2018 but you are using the tokenizer in 2026, new technical terms, product names, and cultural references that emerged after training will be split into small pieces. The word "ChatGPT", for example, would be tokenized by BERT's 2018 vocabulary into something like chat, ##g, ##pt, which captures almost none of its meaning. This vocabulary staleness is a practical problem for long-lived models and requires either periodic retraining of the tokenizer or careful out-of-vocabulary handling.

The training corpus bias also affects how well the tokenizer handles different registers and domains. A tokenizer trained on general web text will have a vocabulary that reflects common web language, with domain-specific technical vocabulary from medicine, law, or specialized sciences being over-split. When using BERT for specialized domains, a common approach is domain-adaptive pre-training with a re-trained tokenizer, precisely to give the vocabulary better coverage of domain-specific terminology.

WordPiece's most lasting contribution is its role in BERT. BERT demonstrated that pretraining transformer models on large text corpora with a masked language modeling objective could produce representations powerful enough to set state-of-the-art results across nearly every NLP benchmark, and WordPiece was the tokenizer that made this possible at a practical vocabulary size.

The choice to use a 30,000-token vocabulary was significant. Earlier neural language models often used vocabularies of 50,000 or more words, with substantial unknown word rates. BERT's WordPiece vocabulary achieves an unknown token rate near zero while keeping the vocabulary small enough that the embedding matrix does not dominate model parameters. This efficiency helped make BERT and its descendants (RoBERTa, DistilBERT, ALBERT) practical to train and deploy.

WordPiece also popularized the ## continuation prefix convention. The practice of explicitly marking subword boundaries in the token stream has influenced tokenizer design well beyond BERT. SentencePiece (which we will cover in a later chapter) uses a different convention (a special prefix to mark word beginnings rather than word continuations), but both approaches solve the same problem: preserving word boundary information within a subword vocabulary so downstream models can reconstruct the original tokenization.

The likelihood-based scoring that distinguishes WordPiece from BPE has influenced the development of the Unigram Language Model tokenizer, which takes this probabilistic framing much further. Where WordPiece uses PMI to choose among candidate merges in a bottom-up process, the Unigram model starts from a large vocabulary and prunes it by maximizing data likelihood, giving even more principled probabilistic tokenization. The two algorithms share the same philosophical foundation: both ask "what vocabulary best explains the training corpus?" and use that question to guide vocabulary construction. WordPiece does so approximately, one merge at a time. The Unigram model does so globally, optimizing over the entire vocabulary simultaneously.

The broader lesson of WordPiece is that the choice of tokenization has downstream consequences for everything the model learns. The subword vocabulary shapes which units receive embeddings, which units share embeddings, and which morphological patterns are visible to the model's attention mechanism. The move from word-level to subword tokenization, enabled by algorithms like WordPiece, was one of the key factors that made transformer models practical: it allowed a single model with a fixed vocabulary to handle open-domain text without drowning in out-of-vocabulary tokens.

WordPiece is a subword tokenization algorithm developed for BERT that differs from Byte Pair Encoding in one defining respect: instead of selecting merges by raw frequency, it uses a PMI-based score that asks whether two tokens appear together more often than their individual frequencies would predict. This produces vocabularies that tend to preserve meaningful morphological units.

Key takeaways from this chapter:

• The merge criterion: WordPiece maximizes the likelihood of the training corpus at each step, selecting the merge with the highest score $P(AB) / (P(A) \cdot P(B))$, which is equivalent to maximizing pointwise mutual information.
• The ## prefix: Continuation tokens (those appearing in the middle or end of a word) are marked with ##, preserving word boundary information in the token stream for downstream tasks.
• Greedy tokenization: At inference time, WordPiece tokenizes by finding the longest matching prefix at each step, requiring only a vocabulary lookup rather than sequential rule application.
• BERT integration: BERT's 30,522-token WordPiece vocabulary achieves near-zero unknown token rates and has enabled state-of-the-art performance across nearly all NLP benchmarks.
• Handling unknown characters: Single characters are always available as fallback tokens. This ensures complete coverage; rare scripts may produce long character-level splits that can exhaust sequence length budgets.
• Limitations: Morphologically rich languages are over-split, the vocabulary becomes stale as language evolves, and the ## convention does not translate cleanly to scripts without whitespace-delimited words.

The next chapter covers the Unigram Language Model tokenizer, which takes the probabilistic foundation introduced by WordPiece and extends it into a fully generative framework that can assign probabilities to all possible segmentations of a word, rather than only the greedy best.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about WordPiece tokenization.