The Vocabulary Problem › The Vocabulary Problem

You've built bag-of-words representations. You've trained word embeddings that capture semantic relationships. You've watched models learn that "king minus man plus woman equals queen." But lurking beneath all of these techniques is a fundamental challenge that every practitioner eventually confronts: what happens when your model encounters a word it has never seen before?

This is the vocabulary problem, and it is more pervasive than it might initially appear. Every time someone coins a new term, makes a typo, uses a technical abbreviation, switches languages mid-sentence, or writes in a language with rich morphology, traditional word-based models falter in a way that is both predictable and hard to fix. The word "ChatGPT" didn't exist before 2022, yet models trained before then need to process it in user queries. The misspelling "reccomend" isn't in any dictionary, yet users type it constantly in product reviews and search queries. The German compound "Bundesausbildungsförderungsgesetz" is a perfectly valid, officially recognized word meaning the federal law governing educational funding, yet it will almost certainly be absent from any vocabulary built from a standard corpus. Each of these cases exposes the same fragility: word-level models treat every surface form as an indivisible atom, and any atom that wasn't seen during training simply doesn't exist.

Think of a fixed word vocabulary as a dictionary locked in a library. The dictionary was compiled at a specific point in time, by specific editors, drawing on specific source texts. Everything outside that dictionary is, from the library's perspective, not a word. But language doesn't care about library hours. New terms enter use every day: product names, scientific discoveries, cultural phenomena, internet slang. The dictionary can't grow without a new edition, and a model's vocabulary can't grow without retraining.

The scale of the problem becomes clearer when you consider the different sources of unknown words. There are the newly coined words that didn't exist when training data was collected. There are the domain-specific terms that appear only in technical literature the training corpus never included. There are the proper nouns, which are both numerous and constantly changing. There are the morphological variants, words formed by applying prefixes and suffixes to known roots in combinations the model didn't see. And there are the orthographic variants: capitalizations, hyphenations, abbreviations, and misspellings. Any real deployment environment generates a steady stream of text from all these categories simultaneously.

The consequences ripple through every downstream task. In machine translation, an unknown source word blocks translation of an entire phrase. In question answering, a medical term the model doesn't recognize can invalidate an otherwise correct reasoning chain. In named entity recognition, a novel organization name collapses to [UNK] and can't be labeled. The vocabulary problem isn't an edge case to be handled gracefully; it's a structural limitation baked into the architecture of word-level systems.

This chapter explores why word-level tokenization breaks down in practice. We'll examine the explosion of vocabulary sizes, the curse of rare words, the special challenges posed by morphologically rich languages and domain-specific text, and the fundamental tension between coverage and efficiency. By understanding these limitations in depth, you'll see clearly why subword tokenization, covered in the following chapters, became essential for modern NLP. The path from word-level models to transformers like BERT and GPT runs directly through the vocabulary problem and its solution.