Byte Pair Encoding › Why BPE Works

Before diving into mechanics, it is worth understanding the intuition behind BPE's design. Every practical engineering decision in the algorithm reflects a deliberate choice about what matters in language representation, and seeing those connections early makes the later details much more natural.

Natural language has structure at multiple scales. Morphemes are the smallest units of meaning: prefixes like "un-" and "re-", roots like "walk" and "play", and suffixes like "-ing", "-tion", and "-er". These units appear repeatedly across thousands of different words. A tokenizer that represents them as single units needs far fewer tokens per sentence than one working character-by-character, yet far fewer vocabulary entries than one trying to enumerate every possible word form. Consider the word family "play", "plays", "player", "playing", "replayed", "unplayable". A character-level model treats each of these as a string of independent characters. A word-level model treats each as an atomic unit with no connection to the others. A subword model can represent all of them using shared tokens for "play", "-er", "-ing", "re-", and "un-", letting the downstream neural network learn that these shared tokens carry related meaning.

BPE discovers these units automatically from data. You do not specify that "ing" or "tion" are important; the algorithm notices they appear frequently and merges them. The result is a vocabulary that reflects the actual distributional patterns of the language it was trained on. For code, it discovers common keywords and operators. For German, it handles compounds. For multilingual corpora, it allocates tokens proportionally to the frequency of each language's patterns. This data-driven quality lets the same algorithm work for any language or domain without modification. You do not need a linguist's analysis of Turkish morphology or an engineer's understanding of Python syntax. You need only a large corpus.

The greediness is a feature, not a limitation. Each merge targets the pair that provides the greatest compression gain at that moment. This greedy schedule produces vocabularies that are empirically close to optimal for downstream tasks, and its simplicity makes training fast and deterministic. Running BPE on a corpus of billions of words takes hours rather than days. The algorithm always produces the same result for the same corpus and target vocabulary size, which is important for reproducibility. And the output, an ordered list of merge rules, is a compact artifact that can be saved, shared, and applied to any new text with no further training.

The key insight is that language compression and language modeling are deeply aligned. The patterns that compress text efficiently, common morphemes, frequent word endings, recurring prefixes, are exactly the patterns that carry consistent meaning across contexts. By optimizing for compression, BPE is indirectly optimizing for linguistic structure. This is why it works better than arbitrary segmentation schemes and why it generalizes so cleanly to languages the algorithm designers never anticipated.