Byte Pair Encoding › Summary

Byte Pair Encoding solves the vocabulary problem by iteratively discovering which character sequences are frequent enough to warrant their own token. Starting from individual characters and applying greedy merge operations, BPE constructs a vocabulary of subword units that provides compact representations for common patterns and graceful decomposition for rare or novel words. Its success comes from a fortunate alignment: optimizing for compression naturally recovers linguistic structure, because the patterns most worth representing compactly are the patterns that carry the most consistent meaning.

The key ideas to take away from this chapter are:

• Training runs in two steps: count adjacent pair frequencies, merge the most frequent pair, repeat until the vocabulary reaches the target size. Each merge captures one frequency pattern from the training corpus.
• Encoding applies merge rules in the order they were learned. The ordering encodes dependency: later rules can only fire after earlier rules have created their expected inputs.
• Decoding is trivial concatenation. BPE is lossless: encoding followed by decoding always reconstructs the original text.
• Vocabulary size is the primary hyperparameter. Larger vocabularies reduce sequence length (lowering compute cost) but increase embedding parameters and may include tokens with insufficient training signal.
• Byte-level BPE extends the approach to operate on raw bytes, guaranteeing universal coverage with no unknown tokens. This is the standard used in GPT-2, GPT-3, and GPT-4.
• Pre-tokenization defines boundaries that merges cannot cross, preventing cross-word artifacts and encoding word-position information in the token representations.
• Domain and language matter. A tokenizer trained on English news articles will tokenize code, medical text, or low-resource languages inefficiently. Specialized models often use specialized tokenizers.

In the next chapter, we will examine WordPiece, BERT's tokenizer. WordPiece modifies the merge criterion: instead of always merging the most frequent pair, it maximizes the likelihood of the training corpus under a unigram language model. This subtle change leads to different vocabulary choices and introduces the ## prefix convention for subword continuations, which you have likely seen when inspecting BERT tokenizer outputs.