Byte Pair Encoding › Limitations and Impact

BPE's practical success should not obscure its limitations. Understanding where BPE falls short helps you make better decisions about when to use it as-is, when to modify it, and when to consider alternatives.

Greedy training is suboptimal. The algorithm always merges the single most frequent pair at each step. This local optimum may not be the global optimum: a pair that is moderately frequent now might enable many more merges in later steps if merged first. Think of it like building a house by always laying the single most available brick, regardless of where it will do the most structural good. The unigram language model tokenizer (covered in the next chapter) addresses this by framing tokenization as probabilistic inference, which allows globally better vocabulary selection. In practice, the gap between BPE and unigram vocabularies is small on standard benchmarks, but the theoretical limitation is real.

Tokenization is not universal. Vocabulary size calibrated for English may fragment other languages much more aggressively. Research has shown that some non-Latin-script languages require four to ten times as many tokens as English to express equivalent content using a BPE tokenizer trained predominantly on English text. Arabic, Thai, and many African languages are particularly disadvantaged because their morphological complexity and script differences mean that few of the learned merges apply. This has equity implications: models cost more to run on those languages and have been observed to perform worse on tasks requiring longer contexts, because the same semantic content occupies many more token positions.

Sensitivity to pre-tokenization. The decision of where to split before applying BPE merges significantly affects what gets tokenized as a single unit. Numbers, dates, URLs, and code all behave differently depending on how the pre-tokenization regex is written. The GPT-2 tokenizer's treatment of leading spaces has occasionally produced surprising tokenization behavior in downstream applications, particularly for sentence-initial tokens. More significantly, the way numbers are split affects arithmetic performance in language models: models that receive each digit as a separate token can more easily learn positional arithmetic than models where multi-digit numbers get fused into opaque tokens.

No natural token boundary at morpheme level. BPE optimizes for compression, not linguistic coherence. Morphologically meaningful units like "pre-", "-tion", and "-ness" emerge when they are frequent enough, but there is no guarantee. Rare morphemes may be split across tokens in ways that make morphological generalization harder for the downstream model. A word like "reconfiguration" might be split into "recon", "figur", "ation" rather than the morphologically meaningful "re", "config", "ur", "ation". The model must learn to combine these tokens into the right meaning, which requires more training examples than if the tokenization had respected morpheme boundaries.

Inconsistent handling of related words. Because BPE applies merges greedily and the order of merges depends on frequency, it is possible for morphologically related words to receive very different tokenizations. "play" might be a single token, "plays" might be "play" + "s", but "played" might be "play" + "ed" or "play" + "ed" or even "pla" + "yed" depending on what merges were learned. This inconsistency can make it harder for models to generalize morphological patterns.

Despite these limitations, BPE's contribution to modern NLP is hard to overstate. Before subword tokenization became standard, multilingual models required either enormous word-level vocabularies or per-language models. BPE made it practical to train a single model on text in dozens of languages, because any word in any language can always be expressed as a sequence of known subword tokens. The compression efficiency means models can use shorter sequences, reducing the quadratic cost of attention. And the stability of the algorithm, its speed, and its lack of linguistic assumptions made it easy to adopt widely.

The algorithms that followed, including WordPiece (BERT) and the unigram language model (SentencePiece), are refinements on BPE's core insight. They change the criterion for choosing which pairs to merge and the inference procedure for encoding, but the fundamental structure of a vocabulary built from increasingly larger subword units remains the same. BPE's lasting influence appears in the models that use it directly and in its role establishing subword tokenization as the universal standard for neural language modeling.