The Vocabulary Problem › Looking Ahead: Subword Tokenization Algorithms

The following chapters explore the algorithms that make subword tokenization work in practice. Each takes a distinct approach to learning which splits to make, and understanding their differences matters for choosing the right one for your application.

Byte Pair Encoding (BPE) starts with individual characters and iteratively merges the most frequent adjacent pairs. The vocabulary grows bottom-up from characters to common subwords to frequent whole words. BPE is simple to implement, efficient to train, and produces consistent results. It is used by GPT-2, GPT-3, RoBERTa, and many other major models. Because it is greedy, BPE always produces the same tokenization for a given text, which simplifies both training and inference.

WordPiece is similar to BPE but uses a likelihood-based criterion for deciding which pairs to merge. Rather than merging the most frequent pair at each step, WordPiece merges the pair that most increases the language model's likelihood over the training corpus. This tends to produce finer-grained splits that capture morphological structure more accurately. BERT and many Google models use WordPiece. Its tokenization is also deterministic, but the vocabulary it learns tends to better align with linguistically meaningful units.

Unigram Language Model tokenization takes a top-down approach that is conceptually different from both BPE and WordPiece. It starts with a large candidate vocabulary and iteratively removes the pieces that least reduce the language model's likelihood over the corpus. The final vocabulary is a subset of the candidates that maximizes coverage. Unigram's key advantage is that it supports probabilistic tokenization: during training, a word can be sampled from multiple valid segmentations, which acts as data augmentation and improves generalization.

SentencePiece is a framework that implements either BPE or Unigram, treating text as raw byte sequences rather than requiring any pre-tokenization step. This makes it truly language-agnostic: it doesn't need word boundaries, whitespace handling, or language-specific rules. SentencePiece is used by T5, ALBERT, XLNet, and many multilingual models. Its language independence makes it particularly valuable for low-resource languages and cross-lingual transfer learning.

All of these algorithms produce two things: a vocabulary of subword units and a procedure for tokenizing new text using that vocabulary. The key insight uniting them is the same: words are not atoms. They can and should be decomposed into smaller, reusable pieces that allow a finite vocabulary to cover an effectively infinite space of surface forms.