The Vocabulary Problem › Limitations of Subword Tokenization

Before we move on, it is worth acknowledging that subword tokenization is not a perfect solution. It resolves the core problems of word-level tokenization, but it introduces its own set of challenges that practitioners must understand and manage.

The first limitation is that subword boundaries do not always align with linguistic morpheme boundaries. BPE and its variants are frequency-based algorithms: they merge character sequences that appear together often, regardless of whether those sequences correspond to meaningful linguistic units. Common words like "the" and "in" appear so frequently that they are kept as single units, while less frequent words get split at positions that may not correspond to any morphological boundary. The word "unnatural" might split as "un" + "natural" (linguistically correct) or as "unn" + "atural" (an artifact of frequency patterns in the training corpus). These non-linguistic splits don't break the model, but they mean the model must learn to work with token boundaries that carry no guaranteed semantic significance.

The second limitation is that the same text can be tokenized differently in different contexts. Standard subword tokenizers are greedy and deterministic: a given word always splits the same way. But this means that related words may receive very different tokenizations. "run" might be a single token, while "running" splits into "run" + "ning," and "runner" splits into "runner" as a single token if it's frequent enough. The model sees these three tokens as partially distinct objects, with no guaranteed sharing between "runner" and "run" + "ner," even though they share the same root. SentencePiece's unigram mode addresses this partially through probabilistic sampling during training, but the basic tension remains.

The third limitation is that subword tokenization is language and script dependent. A tokenizer trained on English text will perform poorly on Arabic or Chinese because the vocabulary it learns is entirely drawn from the character patterns of English. Multilingual models require careful vocabulary construction that allocates sufficient capacity to each language, and the trade-offs are complex: allocating too many tokens to a low-resource language wastes capacity for high-resource languages, while allocating too few degrades performance on the low-resource language. Truly language-agnostic tokenization remains an active research area.

The fourth limitation is a subtle but important one: tokenization affects model behavior in ways that can be hard to predict or control. The way a word is split determines which parts of it are attended to jointly and which are handled as distinct units. Proper names, numbers, and special characters all present challenges. A date like "2024-03-15" might split as "202" + "4" + "-" + "03" + "-" + "15," which is entirely opaque. A chemical formula like "H2O" might split as "H" + "2" + "O," three units with no indication that they form a meaningful compound. These tokenization artifacts can cause surprising model failures on structured data.

Despite these limitations, subword tokenization is a massive improvement over word-level tokenization for practical NLP. The problems it introduces are more subtle and manageable than the hard failures of word-level OOV. And ongoing research into better tokenization algorithms continues to address each of these limitations.