Byte Pair Encoding › Byte Pair Encoding

In the previous chapter, we saw the fundamental tension in text tokenization: word-level models require a massive vocabulary and fail on unseen words, while character-level models sidestep vocabulary issues but produce sequences so long that models struggle to learn long-range dependencies. Both extremes have real costs in memory, compute, and modeling quality. This tension is not just theoretical. When researchers tried to train neural machine translation systems in the early 2010s, they were forced to choose between a fixed vocabulary that silently dropped unknown words and a character-level representation that made it nearly impossible for models to learn useful patterns over long contexts.

Byte Pair Encoding (BPE) threads this needle. It starts from individual characters and iteratively merges the most frequent adjacent pairs into new tokens. After enough merges, common words become single tokens while rare words decompose into smaller meaningful pieces. You get compact representations for frequent language patterns and graceful fallback for novel words, all without any manually curated vocabulary or linguistic knowledge. The algorithm learns what to represent as a single unit directly from the statistics of the corpus.

Think of BPE as building a dictionary by committee from the ground up. Start with every letter as its own word. Then repeatedly promote the most common two-word phrase into a single compound word. After enough promotions, your dictionary contains common syllables, morphemes, and whole words, all discovered without ever telling the algorithm what a morpheme is. The result reflects the actual patterns of the language rather than any preconceived notion of how language works.

BPE was originally a lossless data compression algorithm introduced by Philip Gage in 1994. Its application to neural machine translation by Rico Sennrich, Barry Haddow, and Alexandra Birch in their landmark 2016 ACL paper transformed it into the default subword tokenization method for the field. That paper demonstrated that BPE solved the unknown word problem in translation and improved translation quality compared to word-level models, by allowing the model to learn morphological patterns explicitly. Today, BPE or close variants power the tokenizers in GPT-2, GPT-3, GPT-4, RoBERTa, LLaMA, and dozens of other models. Understanding it is therefore essential background for anyone working with modern language models.

This chapter covers BPE end-to-end: the algorithm's design principles, the step-by-step training procedure, encoding and decoding new text, vocabulary size tradeoffs, byte-level extensions, and a complete from-scratch implementation. We will also look at where BPE falls short, because understanding its limitations helps you make better decisions when choosing or training tokenizers for practical systems.