The Vocabulary Problem › Summary

The vocabulary problem arises from a fundamental mismatch between the infinite productivity of language and the finite capacity of NLP models. This chapter examined several distinct but closely related challenges that together make word-level tokenization inadequate for modern NLP:

• Out-of-vocabulary words plague any fixed-vocabulary system. New words, rare words, typos, proper nouns, and domain-specific terms all collapse to [UNK], losing their meaning entirely. The OOV problem is not an edge case: it occurs constantly in production systems, and its impact scales with how far the deployment domain differs from the training domain.

• Vocabulary size creates an inescapable tradeoff: larger vocabularies reduce OOV rates but consume more memory, slow computation, and suffer from poorly-learned embeddings for rare words. Coverage follows diminishing returns: each additional percentage point of coverage requires exponentially more vocabulary entries. Perfect coverage is unreachable in principle because language is infinitely productive.

• Rare words create a two-tier system: words that make the vocabulary but appear rarely are little better than [UNK], because their embeddings are too poorly trained to be reliable. The minimum frequency cutoff that most systems impose is an admission that word-level representations simply cannot handle the long tail of language.

• Morphologically rich languages make the problem exponentially worse. Agglutinative languages like Finnish and Turkish can form millions of valid word forms from a small set of morphemes, requiring vocabularies an order of magnitude larger than English for equivalent coverage. Word-level tokenization effectively makes multilingual NLP at scale impractical.

• Domain-specific text including source code, scientific writing, and medical records introduces structured tokens that word-level models cannot decompose. Programming identifiers, chemical formulas, and morphologically complex technical terms all appear as opaque unknown atoms.

• Character-level tokenization eliminates OOV but creates excessively long sequences that degrade model performance, reduce effective context length, and require much more training data to achieve equivalent accuracy.

• Subword tokenization resolves all these tensions by breaking words into reusable pieces. A vocabulary of 30,000 to 50,000 subword units can represent any text without OOV, share parameters across morphologically related words, handle novel terms gracefully, and work across multiple languages from a single vocabulary.

Understanding why word-level tokenization fails is the necessary first step before seeing how subword methods succeed. The gap between the problem we've described in this chapter and the solutions provided by BPE, WordPiece, Unigram, and SentencePiece is the gap that made the transformer revolution possible. The following chapters cover each of these algorithms in detail, showing exactly how they learn their vocabularies, how they tokenize new text at inference time, and what the practical differences between them mean for model performance.