WordPiece › Summary

WordPiece is a subword tokenization algorithm developed for BERT that differs from Byte Pair Encoding in one defining respect: instead of selecting merges by raw frequency, it uses a PMI-based score that asks whether two tokens appear together more often than their individual frequencies would predict. This produces vocabularies that tend to preserve meaningful morphological units.

Key takeaways from this chapter:

• The merge criterion: WordPiece maximizes the likelihood of the training corpus at each step, selecting the merge with the highest score $P(AB) / (P(A) \cdot P(B))$, which is equivalent to maximizing pointwise mutual information.
• The ## prefix: Continuation tokens (those appearing in the middle or end of a word) are marked with ##, preserving word boundary information in the token stream for downstream tasks.
• Greedy tokenization: At inference time, WordPiece tokenizes by finding the longest matching prefix at each step, requiring only a vocabulary lookup rather than sequential rule application.
• BERT integration: BERT's 30,522-token WordPiece vocabulary achieves near-zero unknown token rates and has enabled state-of-the-art performance across nearly all NLP benchmarks.
• Handling unknown characters: Single characters are always available as fallback tokens. This ensures complete coverage; rare scripts may produce long character-level splits that can exhaust sequence length budgets.
• Limitations: Morphologically rich languages are over-split, the vocabulary becomes stale as language evolves, and the ## convention does not translate cleanly to scripts without whitespace-delimited words.

The next chapter covers the Unigram Language Model tokenizer, which takes the probabilistic foundation introduced by WordPiece and extends it into a fully generative framework that can assign probabilities to all possible segmentations of a word, rather than only the greedy best.