WordPiece › WordPiece Tokenization: The BERT Subword Algorithm

Subword tokenization took a decisive turn in 2018 when Google published BERT. While Byte Pair Encoding had already shown that splitting words into pieces rather than treating them as atomic units could dramatically reduce vocabulary problems, BERT's authors chose a different algorithm for their tokenizer: WordPiece. On the surface, WordPiece and BPE look similar. Both start from individual characters, both build up a vocabulary by iteratively merging units, and both produce the same kind of subword tokens you recognize in practice: things like play, ##ing, un, ##der, ##stand. But the way they decide which pairs to merge is fundamentally different, and that difference shapes everything from what the vocabulary captures to how the tokenizer behaves on rare words.

As we covered in the BPE chapter, BPE chooses its merges greedily by frequency: whatever pair appears most often in the training corpus gets merged next. WordPiece takes a probabilistic perspective instead. Rather than asking "which pair appears most often?", it asks "which merge increases the likelihood of the training data the most?" These two questions sound similar but lead to vocabularies with different properties. WordPiece tends to prefer merges that capture meaningful morphological units, because those units appear together more often than you would expect from their individual frequencies alone.

Think of the difference this way: BPE is like a hiring manager who promotes whoever has worked the most hours, while WordPiece is like one who promotes whoever adds the most value beyond what could be explained by their general role. The BPE criterion is simple and transparent; the WordPiece criterion is more selective and rewards statistical association between token pairs. A pair of tokens that are both extremely common individually might co-occur often simply because each one appears everywhere, but WordPiece's scoring would not reward that pairing highly. Only pairs whose co-occurrence is informative, exceeding what their individual popularities would predict, climb to the top of WordPiece's merge queue.

morphologically meaningful units tend to have strong statistical associations. The suffix ##ing appears after play far more often than you would predict from knowing only how often play and ##ing each appear independently. That surplus signal is exactly what WordPiece's scoring captures. BPE would only know that "playing" is frequent. WordPiece additionally knows that "play" and "##ing" are unusually attracted to each other, beyond what their separate frequencies would predict.

This chapter covers how WordPiece's likelihood-based merge criterion works, how the greedy tokenization algorithm applies a learned vocabulary to new text, the ## prefix convention that marks continuation subwords, how BERT uses WordPiece in practice, and how to train a WordPiece tokenizer using the HuggingFace tokenizers library. We will also work through a step-by-step numerical example of the training algorithm, examine the limitations of the approach, and understand why BERT's choice of WordPiece mattered for the field of NLP.