The Vocabulary Problem

You've built bag-of-words representations. You've trained word embeddings that capture semantic relationships. You've watched models learn that "king minus man plus woman equals queen." But lurking beneath all of these techniques is a fundamental challenge that every practitioner eventually confronts: what happens when your model encounters a word it has never seen before?

This is the vocabulary problem, and it is more pervasive than it might initially appear. Every time someone coins a new term, makes a typo, uses a technical abbreviation, switches languages mid-sentence, or writes in a language with rich morphology, traditional word-based models falter in a way that is both predictable and hard to fix. The word "ChatGPT" didn't exist before 2022, yet models trained before then need to process it in user queries. The misspelling "reccomend" isn't in any dictionary, yet users type it constantly in product reviews and search queries. The German compound "Bundesausbildungsförderungsgesetz" is a perfectly valid, officially recognized word meaning the federal law governing educational funding, yet it will almost certainly be absent from any vocabulary built from a standard corpus. Each of these cases exposes the same fragility: word-level models treat every surface form as an indivisible atom, and any atom that wasn't seen during training simply doesn't exist.

Think of a fixed word vocabulary as a dictionary locked in a library. The dictionary was compiled at a specific point in time, by specific editors, drawing on specific source texts. Everything outside that dictionary is, from the library's perspective, not a word. But language doesn't care about library hours. New terms enter use every day: product names, scientific discoveries, cultural phenomena, internet slang. The dictionary can't grow without a new edition, and a model's vocabulary can't grow without retraining.

The scale of the problem becomes clearer when you consider the different sources of unknown words. There are the newly coined words that didn't exist when training data was collected. There are the domain-specific terms that appear only in technical literature the training corpus never included. There are the proper nouns, which are both numerous and constantly changing. There are the morphological variants, words formed by applying prefixes and suffixes to known roots in combinations the model didn't see. And there are the orthographic variants: capitalizations, hyphenations, abbreviations, and misspellings. Any real deployment environment generates a steady stream of text from all these categories simultaneously.

The consequences ripple through every downstream task. In machine translation, an unknown source word blocks translation of an entire phrase. In question answering, a medical term the model doesn't recognize can invalidate an otherwise correct reasoning chain. In named entity recognition, a novel organization name collapses to [UNK] and can't be labeled. The vocabulary problem isn't an edge case to be handled gracefully; it's a structural limitation baked into the architecture of word-level systems.

This chapter explores why word-level tokenization breaks down in practice. We'll examine the explosion of vocabulary sizes, the curse of rare words, the special challenges posed by morphologically rich languages and domain-specific text, and the fundamental tension between coverage and efficiency. By understanding these limitations in depth, you'll see clearly why subword tokenization, covered in the following chapters, became essential for modern NLP. The path from word-level models to transformers like BERT and GPT runs directly through the vocabulary problem and its solution.

The out-of-vocabulary (OOV) problem is the most immediate and visible failure mode of word-level models. Any fixed vocabulary, no matter how large, will encounter words it has never seen at inference time. This isn't a pathological edge case: it happens in virtually every production deployment, often dozens of times per sentence in technical or multilingual text.

The fundamental reason is simple: language is infinite, but vocabulary is finite. Natural language allows speakers to create new words freely. A vocabulary compiled from any corpus, however large, is a snapshot of language at a particular moment and in a particular domain. The moment you deploy that vocabulary on new text, from a new time period, a new domain, or a new set of users, the gap between what your vocabulary covers and what users write immediately becomes apparent.

Before we examine the mechanics of the OOV problem, it helps to understand exactly what "vocabulary" means in a word-level NLP system. When training, you collect all words that appear above some frequency threshold in your corpus. Each word gets a unique integer index. The embedding matrix has one row per word; the output projection layer has one column per word. Every component of the model that touches word identity is sized to exactly this vocabulary. There is no room for new words without resizing these tensors and retraining. The vocabulary is a list baked into the architecture itself.

Consider a sentiment analysis model trained on movie reviews. It learned representations for words like "excellent," "boring," and "cinematography." The embedding for "excellent" is close to "wonderful" and "fantastic." The model learned that these words reliably predict positive sentiment. Now imagine deploying this model and encountering the review: "This movie was amazeballs! The CGI was unreal."

The word "amazeballs" is almost certainly not in the training vocabulary. It entered common use in the early 2010s as internet slang expressing enthusiastic approval. What should the model do? The word carries strong positive sentiment, but the model has no entry for it. Three classic approaches exist, and none of them is satisfactory:

1. Replace with a special [UNK] token: The model treats all unknown words identically, losing important information. "Amazeballs" and "terrible" both become [UNK], erasing the distinction between positive and negative sentiment. The model has neededly been blinded to a key signal in the text.

2. Skip the word entirely: The sentence becomes "This movie was ! The CGI was unreal." Syntactic structure is partially preserved, but the most sentiment-laden word in the sentence is gone. Any analysis of this sentence will be based on incomplete information.

3. Attempt approximate matching: Maybe "amazeballs" is similar to "amazing"? This requires additional infrastructure such as edit-distance lookups or phonetic matching, it's computationally expensive, and it often fails for truly novel words where there is no close neighbor in the vocabulary.

The [UNK] approach is overwhelmingly the most common choice in practice because it requires no extra computation and is simple to implement. But it creates a black hole in your model's understanding. Any word outside the vocabulary collapses to a single undifferentiated representation. The model cannot distinguish between "amazeballs" (very positive), "meh" (neutral), and "unwatchable" (very negative) once they all become [UNK]. The key insight is that [UNK] doesn't just remove information about one word: it actively introduces noise by conflating semantically opposite concepts under a single token.

Let's quantify how serious this problem is in practice. We'll build a vocabulary from one text corpus and measure how many words from a different corpus fall outside it. This domain-shift scenario is extremely common: a model trained on news articles is deployed to process social media posts, or a model trained on formal academic text must handle informal customer support conversations.

Even in this small demonstration, a substantial fraction of test tokens are unknown to the vocabulary built from classic literature. Words like "smartphone," "oled," "bluetooth," and "gpu" are completely absent because they represent modern technology that didn't exist when those texts were written. This illustrates a critical pattern: domain shift between training and deployment data creates OOV problems even when vocabulary sizes seem entirely adequate for the training domain. The same failure occurs in every deployment scenario where the text distribution shifts: a medical model encountering new drug names, a legal model seeing recently enacted legislation, a social media model encountering today's trending slang.

The OOV problem reveals that word-level models treat language as a closed system, a fixed catalog of known atoms. Real language is an open system where new expressions are continuously coined. No training corpus, however extensive, captures all the text that will ever be written. In real applications, the OOV problem never goes away because language continuously evolves and any fixed vocabulary grows stale from the moment training ends.

The OOV problem stems from a fundamental statistical property of natural language: word frequencies follow Zipf's law. A small number of words appear extremely frequently, while an enormous number of words appear rarely or only once. This isn't a quirk of any particular language or corpus; it has been observed across hundreds of languages and text types, from Ancient Greek to modern Twitter.

Why does Zipf's law matter for the vocabulary problem? Because it means the frequency distribution is extremely skewed. The top 100 words typically cover about 50% of all tokens in English text. The top 1,000 words cover about 75%. But the vocabulary of a typical adult speaker contains 30,000 to 50,000 words, most of which appear infrequently. There are hundreds of thousands of words that appear only once or twice per million tokens, and the long tail stretches essentially to infinity when you account for new coinages, proper nouns, technical terms, and spelling variants.

The log-log plot makes the Zipfian structure immediately visible. The top few words appear millions of times; the majority of vocabulary items appear only a handful of times each. The straight line on the log-log plot is the hallmark of a power law: frequency drops off smoothly and steeply as rank increases. The "long tail" region to the right of the chart contains most of the unique word types, but each individual word there is so rare that any reasonably sized corpus will miss many of them entirely.

The key insight is that the long tail is not just large, it is also unbounded. Every day, new words enter English from technology, science, culture, and the internet. Every new product launch creates new proper nouns. Every emerging field generates new technical terms. The tail keeps growing. A vocabulary fixed at training time is racing against an exponentially expanding frontier of new words, and it cannot win. This is not a temporary problem that will go away as we collect more training data: it is a structural consequence of how human language works.

Managing the OOV problem requires making a difficult choice about vocabulary size. This choice sits at the heart of word-level system design, and every option comes with painful tradeoffs. Understanding these tradeoffs precisely is essential for appreciating why subword methods were such a breakthrough.

Think of vocabulary size as a dial with no good setting. Turn it down and you have a fast, memory-efficient system that understands very little of what users write. Turn it up and you have broader coverage at the cost of massive memory consumption, slow computation, and paradoxically worse embeddings for the rare words you're trying to include. The tradeoff isn't just technical: it reflects a deeper tension between the finite nature of computational resources and the infinite productivity of language.

How large should your vocabulary be? This question reveals a fundamental tension in NLP system design that practitioners grapple with every time they train a new model.

A small vocabulary is computationally efficient. The embedding matrix, output projection layer, and any word-based operations all scale linearly with vocabulary size. A vocabulary of 10,000 words means 10,000 embeddings to store and 10,000 output classes for word prediction. Model parameters are proportional to vocabulary size: halving the vocabulary roughly halves the parameter count for embedding-heavy architectures. Training is faster, inference is faster, and memory consumption is manageable.

But a small vocabulary means a high OOV rate. Users will constantly encounter the [UNK] token, degrading model performance across every task. Text from specialized domains, from different time periods, or from users with non-standard spelling will be heavily affected. The model becomes brittle: it performs well on text similar to its training data and fails unpredictably on anything else.

A large vocabulary reduces OOV rates but introduces its own set of serious problems:

• Memory explosion: Each word needs an embedding vector. With 300-dimensional float32 embeddings, each word requires 1,200 bytes. At 1 million words, the embedding matrix alone consumes 1.2 GB. At 10 million words, it consumes 12 GB, often exceeding the memory of a single GPU and making distributed training a necessity.
• Sparse gradients: Rare words appear infrequently during training, so their embeddings receive few gradient updates and remain poorly learned. A word appearing 3 times in a billion-token corpus has an embedding that's essentially random noise, just as useless as [UNK] but consuming memory.
• Computational cost: The final layer of a language model computes a softmax distribution over the entire vocabulary to predict the next word. This operation scales linearly with vocabulary size. Computing softmax over 1 million classes requires a million floating-point exponentials and divisions per prediction step, which is prohibitively expensive during both training and inference without special approximations.
• Diminishing returns: Each word added to the vocabulary covers a progressively less frequent token. The 50,001st word in your vocabulary might appear once per 10 million tokens, contributing almost nothing to practical coverage while adding the same per-word cost as "the."

Let's visualize how vocabulary size affects corpus coverage, using the power-law structure of Zipf's law to model coverage as a function of vocabulary size.

The curve reveals a sobering truth: achieving high coverage requires exponentially larger vocabularies, but the returns diminish sharply. Going from 75% to 90% coverage means multiplying your vocabulary by 10, from roughly 1,000 words to 10,000. Going from 90% to 96% requires multiplying by 5 again, from 10,000 to 50,000 words. Going from 96% to 98% requires doubling again to 100,000 words. Each additional percentage point of coverage is more expensive than the last. And 100% coverage is essentially impossible because language is infinitely productive, with new words entering use every day. The curve never reaches 100%; it asymptotically approaches it while the vocabulary grows without bound.

Let's examine actual vocabulary sizes from popular NLP resources to understand the practical scale of this problem and the resulting memory costs.

The memory requirements tell a stark story. Basic English's 850 words need less than a megabyte for embeddings, while large pre-trained word vectors like Word2Vec or GloVe contain millions of entries requiring multiple gigabytes. Each additional word in the vocabulary adds exactly $300 \times 4 = 1{,}200$ bytes (1.2 KB) for its 300-dimensional float32 embedding vector. At 3 million words, the embedding matrix alone consumes 3.4 GB, before any model weights. The Google Web 1T corpus, with over 13 million unique word forms, would require more than 14 GB just for the embedding table. This creates a practical ceiling on vocabulary size and forces practitioners to choose between coverage and computational efficiency.

The numbers in this table represent real engineering constraints. Modern GPU memory tops out at 40 to 80 GB for the most expensive hardware. A 14 GB embedding table leaves very little room for the rest of the model, attention layers, feed-forward networks, and batch activations during training. In practice, most word-level models cap their vocabularies at 50,000 to 100,000 words precisely because larger vocabularies become difficult to fit on a single accelerator, and gradient updates for rare words in multi-GPU setups require expensive communication. Vocabulary size is both a coverage decision and an infrastructure constraint.

Even when rare words make it into the vocabulary, they face a different and subtler problem: they simply cannot be learned well. This is perhaps the most insidious aspect of word-level tokenization because it creates a two-tier system where frequent words have rich, reliable representations and rare words have poor, unreliable ones, and the boundary between them is blurry.

To understand why this happens, recall how word embeddings are learned. Each word's vector representation is updated based on the contexts in which it appears. Each time a word occurs in training data, its embedding receives a gradient signal telling it to be more similar to its context words. A word appearing 100,000 times receives 100,000 such updates across training, each gradually refining the embedding to better capture the word's semantic neighborhood. A word appearing 10 times receives only 10 gradient updates, and its embedding remains close to its random initialization.

The problem comes from both the quantity of updates and their diversity. A frequent word is observed in many different contexts: formal and informal sentences, different topics, different syntactic positions. This diversity of contexts gives the model a varied signal about the word's meaning. A rare word might appear in only one or two semantic contexts, so even if it appears a dozen times, those appearances may all be similar. This provides a narrow and potentially unrepresentative signal.

The result is that rare word embeddings are poorly learned. They may be nearly random vectors, barely moved from their random initialization. This isn't a limitation of the learning algorithm: it is a fundamental consequence of having too little signal. No learning algorithm can extract information that isn't present in the data, and a word that appears 5 times simply doesn't provide enough context to reliably determine its meaning relative to all other words.

Think of learning word meanings the way a child learns vocabulary: through repeated exposure in varied contexts. You learn what "serendipity" means because you've heard it used to describe unexpected good fortune across dozens of situations: stories, movies, casual conversation. If you heard the word only once, in a single sentence, your understanding would be vague and unreliable. Word embedding training faces the exact same constraint, just formalized mathematically.

This creates a vicious cycle. Rare words have poor embeddings, which means they contribute little reliable signal to downstream tasks, which means task-specific fine-tuning can't improve them because there isn't enough data. The model learns to ignore rare words or treat them as noise, which reinforces their poor representation. Even if you include a rare word in the vocabulary, you often get little practical benefit over [UNK] because the embedding is so poorly initialized that the model can't make good use of it.

To avoid polluting the vocabulary with poorly-learned embeddings, most word embedding methods impose a minimum frequency cutoff. Words appearing fewer than, say, 5 times are excluded from the vocabulary entirely and mapped to [UNK]. The rationale is that a word seen only once or twice doesn't have enough signal to train a useful embedding, so it's better to exclude it and keep the model clean than to include it with a near-random vector that might actively mislead downstream tasks.

Word2Vec's min_count parameter defaults to 5 for this reason. GloVe's co-occurrence matrix construction ignores words below a frequency threshold. FastText, which we'll discuss shortly, is one of the few word-level methods that partially addresses this problem by using character n-grams. But for standard word embedding methods, the minimum frequency cutoff is a necessary compromise.

The table makes the tradeoff concrete. Setting min_count=1 keeps all 13 words, but entries like "linformer" (appearing only once) will have nearly random embeddings that add noise rather than signal. Raising the cutoff to min_count=5 excludes only the three rarest words, "linformer," "longformer," and "reformer," while improving embedding quality for everything that remains. At min_count=100, only 8 words survive the cutoff. In practice, cutoffs of 5 to 10 are common across major toolkits, but every excluded word becomes OOV at inference time, replaced by [UNK] with no semantic content.

The key insight is that the frequency cutoff is not a solution to the rare word problem: it is an admission of defeat. You're not fixing the problem; you're acknowledging that word-level representations can't handle rare words and choosing to discard them rather than represent them badly. This is a perfectly reasonable engineering decision given the constraints, but it means that rare words, which include many of the most semantically specific and important terms in technical text, simply don't exist for word-level models.

The OOV and vocabulary explosion problems are particularly severe for morphologically complex languages. English has relatively simple morphology: most words have a small number of inflected forms (walk, walks, walked, walking) and a moderate number of derivational forms (walk, walker, walkable, unwalkable). The entire set of surface forms for a given English root is typically fewer than a dozen entries. Other languages work fundamentally differently, and word-level tokenization fails them much more severely.

Think of morphology as a word-building system. Languages differ in how much work their morphology does. English offloads much of its grammatical information to separate words: "I will have been walking" uses five separate word tokens. Turkish and Finnish pack the same information into a single word through a process called agglutination, stacking suffix after suffix onto a root. German builds compound nouns by directly concatenating shorter words, creating new entries that can stretch to extraordinary lengths. For a word-level model, each surface form is a separate atom, regardless of how much shared structure it contains with related forms.

Agglutinative languages like Finnish, Turkish, and Hungarian build words by stacking morphemes, each carrying a distinct grammatical meaning. The morphemes are relatively transparent and consistent: a given suffix always contributes the same meaning when attached to any root. A single Finnish word might encode subject, object, tense, aspect, mood, location, and possession all at once. The word "talossanikinko" packs "in my house too, I wonder?" into a single token from the root "talo" (house) plus four suffixes.

German is famous for compound nouns. German permits, and indeed regularly practices, the creation of long compound words by directly concatenating shorter words without spaces. The word "Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz" is a real, officially used term meaning a law about the delegation of supervisory tasks for beef labeling. Such compounds are regularly coined by combining existing words when naming new concepts, and each new combination is a distinct vocabulary entry for a word-level tokenizer.

Each of these word forms would require its own vocabulary entry under word-level tokenization, even though a native speaker immediately parses the components and understands the meaning by composing the parts. The Finnish suffix chain, "-ssa" for locative case (in a place), "-ni" for first-person possessive (my), "-kin" for the additive particle (also, too), "-ko" for question marker, is completely predictable and transparent to a speaker of Finnish. A child learning Finnish doesn't memorize each word form separately; they learn the morphological rules and apply them productively. But a word-level model that sees each surface form as an independent atom cannot exploit this structure at all.

The same transparency applies to German compounds. A German speaker encountering "Handschuhmacherei" for the first time immediately understands it: a place where Handschuhe (gloves) are made. The components "Hand" (hand), "Schuh" (shoe, a poetic name for glove since gloves are "shoes for the hand"), "macher" (maker), and "-ei" (workshop or trade) are all known words. The compound is novel but completely interpretable. For a word-level model, it's simply absent from the vocabulary.

The combinatorial nature of morphology causes vocabulary explosion at a rate that word-level systems cannot possibly keep up with. Where English might need 50,000 common word forms for adequate coverage, Turkish or Finnish might need hundreds of thousands of valid forms, and the theoretical space of grammatically correct forms is much larger still.

To make this concrete: Turkish has roughly 30 nominal case suffixes, 4 number markers, 8 possessive markers, and additional particles, all of which can combine. Just for nouns, the number of theoretically valid surface forms from a single root runs into the hundreds. For a language with 50,000 root words, the total surface form space is in the millions, most of which will appear only rarely in any individual corpus. No vocabulary of practical size can cover this space.

The numbers drive the point home. A Turkish model trained with a word-level vocabulary needs roughly 8 times more entries than an equivalent English model to achieve the same coverage. This multiplier is not hypothetical: empirical studies of large Turkish text corpora confirm that unique surface forms grow much faster than for English as corpus size increases. The gap is also compounding: a model trained on a small Turkish corpus misses rare words and entire productive morphological paradigms, because even common roots may never appear in all their grammatical forms within a modestly sized training set.

Any model trained across multiple languages faces a particularly acute version of this problem. A multilingual model covering English, Turkish, Finnish, German, and five other languages needs to handle the vocabulary requirements of all of them simultaneously. The combined vocabulary for 10 languages under a word-level scheme could easily exceed 2 to 3 million entries, with the attendant memory and computational costs. This was a major practical obstacle to multilingual NLP before subword methods arrived.

Beyond everyday language variation, NLP systems increasingly face the challenge of processing technical text: source code, scientific papers, medical records, legal documents. Each of these domains introduces vocabulary that general-purpose models have never encountered, and the structure of the unknown tokens differs substantially from everyday OOV words.

Unlike new slang or neologisms, which at least follow the phonological patterns of the language, technical vocabulary often violates standard word formation rules entirely. Gene names like BRCA1 and TP53 are alphanumeric. Chemical formulas like C6H12O6 mix letters, numbers, and subscripts. Programming identifiers like XMLHttpRequest combine acronyms and camelCase. These tokens don't just miss the vocabulary: they wouldn't benefit much from the [UNK] treatment because their structure contains meaningful information that a word-level model cannot exploit.

Consider what happens when a general NLP model encounters source code. Variable names, function names, and identifiers are created by developers using essentially arbitrary character combinations, and they follow conventions that are entirely distinct from natural language word formation. Different developers and codebases use different naming conventions, all of which appear as distinct surface forms to a word-level tokenizer:

• CamelCase: getUserById, XMLHttpRequest, processPayment
• Snake case: get_user_by_id, calculate_total_price
• Screaming snake case: MAX_RETRY_COUNT, API_BASE_URL
• Special tokens: __init__, self.model, @property

The same underlying concept, a function that retrieves a user by their identifier, is expressed as entirely different surface forms depending on the developer's convention. getUserById and get_user_by_id are synonymous, but a word-level model sees them as two completely different, probably unknown tokens. This is a representation problem as well as an OOV problem. Even if both tokens were in the vocabulary, the model would have no way to know they refer to the same concept.

Scientific text uses morphologically complex terms built from Greek and Latin roots that follow systematic patterns. "Phosphofructokinase" is a long word, but it decomposes cleanly: "phospho-" refers to a phosphate group, "fructo-" refers to fructose, and "-kinase" identifies the protein class. A biochemist encountering the term "phosphoglycerokinase" for the first time immediately understands the general category: a kinase acting on a phosphoglycerol substrate. A word-level model sees it as an unknown token with no relationship to any other term.

Medical text combines both patterns: complex Latinate terms ("thrombocytopenia," meaning a deficiency of blood platelets; "atherosclerosis," meaning hardening of arteries due to plaque buildup) alongside cryptic abbreviations ("prn" for "as needed," "bid" for "twice daily," "qid" for "four times daily"). Neither category is well-handled by word-level models.

The decompositions reveal an important insight. Both getUserById and get_user_by_id contain the same four common words ("get", "user", "by", "id"), each of which almost certainly appears in any general vocabulary. But the combined identifier is almost certainly OOV in its raw form. The camelCase splitter also handles acronyms correctly, keeping "XML" and "HTML" together rather than splitting each uppercase letter individually, which preserves semantically meaningful units.

This suggests a general and powerful principle: complex tokens often contain recognizable sub-parts, and a tokenization scheme that can identify and decompose those parts intelligently would dramatically reduce OOV rates without requiring a massive vocabulary. The model could learn that "get" followed by a noun-like element describes a retrieval operation, regardless of whether that noun is "user," "order," "product," or any other entity type. The insight generalizes far beyond code: medical terms built from Greek roots, chemical names, and morphologically complex words in any language all exhibit the same structure. That intuition is precisely what motivates subword tokenization, and we'll see how BPE and WordPiece turn this intuition into an algorithm in the following chapters.

At first glance, character-level tokenization seems to resolve the vocabulary problem elegantly. If you tokenize at the character level, there are only a few hundred possible tokens for any given writing system, and OOV is literally impossible for any text written in a supported script. No matter what word a user types, no matter how novel or misspelled, it can be represented as a sequence of characters that are all in the vocabulary. The vocabulary explodes to millions of word forms with word-level tokenization; with character-level tokenization, you need only 26 letters for English, plus some punctuation. Problem solved?

Character-level tokenization does not eliminate the issue; it solves the OOV problem by shifting it to a different level: the model must now learn to compose characters into meaningful units. Characters themselves carry little semantic information. The letter "c" doesn't mean anything until it combines with other letters into words and morphemes. A model operating on characters must learn, from scratch and implicitly, that the sequence "c-a-t" represents the same concept as "c-a-t-s" plus a plural marker, and that "c-h-a-t" is a different concept entirely. All the linguistic structure that word-level models get for free by using words as the basic unit must be rediscovered from character sequences.

The problem is not just conceptual: it is also computational. A single word like "transformers" becomes 12 characters. A typical sentence of 15 words might contain 75 to 100 characters including spaces. Modern transformer models have a limited context window, measured in tokens, so longer token sequences mean less text fits in memory. A 512-token context window can hold roughly 500 words of text at word-level tokenization, but only 100 to 125 words at character-level tokenization. Reducing the effective context length by a factor of 4 to 5 is a severe degradation for tasks requiring long-range reasoning like summarization, document understanding, and multi-turn dialogue.

Empirically, character-level models require more parameters and training data than word-level models to achieve equivalent performance on most tasks. Studies have shown that character-level language models need roughly 5 to 10 times more training tokens to match the perplexity of word-level models on the same text domain, because each prediction step carries less information and the model must expend capacity learning to compose characters into meaningful patterns. The information is technically present in the character sequence, but it is harder to extract and requires more model capacity and training to do so.

Subword tokenization sits in the sweet spot between words and characters. Subwords are longer than characters, so each token carries more semantic content and sequences are shorter. They are shorter than words, so novel terms can be represented through composition without requiring vocabulary entries for every surface form. A vocabulary of 30,000 to 50,000 subword units can represent virtually any text without OOV, at a sequence length overhead of only 10 to 30% compared to word-level tokenization. The key insight is that subwords capture the natural units of language: morphemes, common prefixes and suffixes, high-frequency word fragments that recur across many contexts.

All the challenges we've examined point toward a common solution: stop treating words as indivisible atoms. If we break words into smaller, reusable pieces, we gain several important properties simultaneously. This unification is what makes subword tokenization so powerful: it doesn't just solve one problem, it dissolves the entire cluster of problems that plagued word-level tokenization.

Think of subword units as the natural vocabulary of morphology. Human languages are built on the principle of double articulation: a finite set of meaningless sounds (phonemes) combines to form a finite set of meaningful minimal units (morphemes), which combine to form an unbounded set of words and sentences. Subword tokenization approximates this structure without requiring linguistic expertise: it learns, from data, which character sequences are worth treating as units.

The benefits of moving from words to subwords are:

1. No OOV: Any word, however novel, can be decomposed into known subword pieces. At worst, each character is its own token. The OOV problem disappears structurally, not because the vocabulary is large enough to cover everything, but because the decomposition can always fall back to smaller pieces.

2. Parameter sharing: The piece "un-" learned from "undo" transfers to "unfair," "unhappy," "unclear," and any future "un-" word. Every time the model processes a word containing a known subword, it draws on everything it has learned from all other contexts where that subword appeared. This is the key mechanism: subword representations generalize across words in a way that whole-word representations cannot.

3. Manageable vocabulary: Thousands of subword units can generate millions of word forms. A vocabulary of 32,000 pieces can represent essentially all of English, most other European languages, and a substantial fraction of any language written in Latin script, without any OOV tokens and without the memory burden of millions of word entries.

4. Morphological decomposition: Complex words naturally break at morphological boundaries. "Preprocessing" splits into "pre," "process," and "ing," three meaningful units whose combination correctly suggests the meaning. This decomposition is learned from data, not encoded by hand, but it often aligns surprisingly well with linguistic morpheme boundaries.

5. Cross-lingual transfer: Because subword methods learn language-independent character patterns, a multilingual model can share vocabulary across languages that use the same script. Many words in related languages share prefixes, roots, or suffixes, and subword models can exploit this structure without any explicit cross-lingual supervision.

The decomposition table illustrates the key property: diverse words share subword pieces. "Unhappiness" and "unbelievable" both contain "un-", so learning that prefix from one word immediately benefits processing the other. "Preprocessing" and any other "-ing" word share that suffix. Novel words like "ChatGPT" and "COVID19" split into segments that, while not morphologically meaningful in the traditional sense, are learnable patterns the model can encode. The model might learn that "GPT" tends to appear in contexts related to language models, so any word containing "GPT" inherits that context. Every piece learned from one word is immediately available for composing any other word that contains it.

This approach avoids [UNK] and shares parameters through subword reuse. Parameter sharing through subword reuse means that the model's effective training data for any given morpheme is the sum of its occurrences across all words rather than occurrences of any particular word form. "Un-" appears in "undo," "unfair," "unhappy," "unreal," "unbelievable," and hundreds of other words. A subword model trained on all of these forms builds a rich representation of what "un-" means, and applies that representation when it encounters a new "un-" word. This is fundamentally different from word-level models, where "undo" and "unfair" are entirely separate entries with no shared parameters.

Before we move on, it is worth acknowledging that subword tokenization is not a perfect solution. It resolves the core problems of word-level tokenization, but it introduces its own set of challenges that practitioners must understand and manage.

The first limitation is that subword boundaries do not always align with linguistic morpheme boundaries. BPE and its variants are frequency-based algorithms: they merge character sequences that appear together often, regardless of whether those sequences correspond to meaningful linguistic units. Common words like "the" and "in" appear so frequently that they are kept as single units, while less frequent words get split at positions that may not correspond to any morphological boundary. The word "unnatural" might split as "un" + "natural" (linguistically correct) or as "unn" + "atural" (an artifact of frequency patterns in the training corpus). These non-linguistic splits don't break the model, but they mean the model must learn to work with token boundaries that carry no guaranteed semantic significance.

The second limitation is that the same text can be tokenized differently in different contexts. Standard subword tokenizers are greedy and deterministic: a given word always splits the same way. But this means that related words may receive very different tokenizations. "run" might be a single token, while "running" splits into "run" + "ning," and "runner" splits into "runner" as a single token if it's frequent enough. The model sees these three tokens as partially distinct objects, with no guaranteed sharing between "runner" and "run" + "ner," even though they share the same root. SentencePiece's unigram mode addresses this partially through probabilistic sampling during training, but the basic tension remains.

The third limitation is that subword tokenization is language and script dependent. A tokenizer trained on English text will perform poorly on Arabic or Chinese because the vocabulary it learns is entirely drawn from the character patterns of English. Multilingual models require careful vocabulary construction that allocates sufficient capacity to each language, and the trade-offs are complex: allocating too many tokens to a low-resource language wastes capacity for high-resource languages, while allocating too few degrades performance on the low-resource language. Truly language-agnostic tokenization remains an active research area.

The fourth limitation is a subtle but important one: tokenization affects model behavior in ways that can be hard to predict or control. The way a word is split determines which parts of it are attended to jointly and which are handled as distinct units. Proper names, numbers, and special characters all present challenges. A date like "2024-03-15" might split as "202" + "4" + "-" + "03" + "-" + "15," which is entirely opaque. A chemical formula like "H2O" might split as "H" + "2" + "O," three units with no indication that they form a meaningful compound. These tokenization artifacts can cause surprising model failures on structured data.

Despite these limitations, subword tokenization is a massive improvement over word-level tokenization for practical NLP. The problems it introduces are more subtle and manageable than the hard failures of word-level OOV. And ongoing research into better tokenization algorithms continues to address each of these limitations.

The following chapters explore the algorithms that make subword tokenization work in practice. Each takes a distinct approach to learning which splits to make, and understanding their differences matters for choosing the right one for your application.

Byte Pair Encoding (BPE) starts with individual characters and iteratively merges the most frequent adjacent pairs. The vocabulary grows bottom-up from characters to common subwords to frequent whole words. BPE is simple to implement, efficient to train, and produces consistent results. It is used by GPT-2, GPT-3, RoBERTa, and many other major models. Because it is greedy, BPE always produces the same tokenization for a given text, which simplifies both training and inference.

WordPiece is similar to BPE but uses a likelihood-based criterion for deciding which pairs to merge. Rather than merging the most frequent pair at each step, WordPiece merges the pair that most increases the language model's likelihood over the training corpus. This tends to produce finer-grained splits that capture morphological structure more accurately. BERT and many Google models use WordPiece. Its tokenization is also deterministic, but the vocabulary it learns tends to better align with linguistically meaningful units.

Unigram Language Model tokenization takes a top-down approach that is conceptually different from both BPE and WordPiece. It starts with a large candidate vocabulary and iteratively removes the pieces that least reduce the language model's likelihood over the corpus. The final vocabulary is a subset of the candidates that maximizes coverage. Unigram's key advantage is that it supports probabilistic tokenization: during training, a word can be sampled from multiple valid segmentations, which acts as data augmentation and improves generalization.

SentencePiece is a framework that implements either BPE or Unigram, treating text as raw byte sequences rather than requiring any pre-tokenization step. This makes it truly language-agnostic: it doesn't need word boundaries, whitespace handling, or language-specific rules. SentencePiece is used by T5, ALBERT, XLNet, and many multilingual models. Its language independence makes it particularly valuable for low-resource languages and cross-lingual transfer learning.

All of these algorithms produce two things: a vocabulary of subword units and a procedure for tokenizing new text using that vocabulary. The key insight uniting them is the same: words are not atoms. They can and should be decomposed into smaller, reusable pieces that allow a finite vocabulary to cover an effectively infinite space of surface forms.

The vocabulary problem arises from a fundamental mismatch between the infinite productivity of language and the finite capacity of NLP models. This chapter examined several distinct but closely related challenges that together make word-level tokenization inadequate for modern NLP:

• Out-of-vocabulary words plague any fixed-vocabulary system. New words, rare words, typos, proper nouns, and domain-specific terms all collapse to [UNK], losing their meaning entirely. The OOV problem is not an edge case: it occurs constantly in production systems, and its impact scales with how far the deployment domain differs from the training domain.

• Vocabulary size creates an inescapable tradeoff: larger vocabularies reduce OOV rates but consume more memory, slow computation, and suffer from poorly-learned embeddings for rare words. Coverage follows diminishing returns: each additional percentage point of coverage requires exponentially more vocabulary entries. Perfect coverage is unreachable in principle because language is infinitely productive.

• Rare words create a two-tier system: words that make the vocabulary but appear rarely are little better than [UNK], because their embeddings are too poorly trained to be reliable. The minimum frequency cutoff that most systems impose is an admission that word-level representations simply cannot handle the long tail of language.

• Morphologically rich languages make the problem exponentially worse. Agglutinative languages like Finnish and Turkish can form millions of valid word forms from a small set of morphemes, requiring vocabularies an order of magnitude larger than English for equivalent coverage. Word-level tokenization effectively makes multilingual NLP at scale impractical.

• Domain-specific text including source code, scientific writing, and medical records introduces structured tokens that word-level models cannot decompose. Programming identifiers, chemical formulas, and morphologically complex technical terms all appear as opaque unknown atoms.

• Character-level tokenization eliminates OOV but creates excessively long sequences that degrade model performance, reduce effective context length, and require much more training data to achieve equivalent accuracy.

• Subword tokenization resolves all these tensions by breaking words into reusable pieces. A vocabulary of 30,000 to 50,000 subword units can represent any text without OOV, share parameters across morphologically related words, handle novel terms gracefully, and work across multiple languages from a single vocabulary.

Understanding why word-level tokenization fails is the necessary first step before seeing how subword methods succeed. The gap between the problem we've described in this chapter and the solutions provided by BPE, WordPiece, Unigram, and SentencePiece is the gap that made the transformer revolution possible. The following chapters cover each of these algorithms in detail, showing exactly how they learn their vocabularies, how they tokenize new text at inference time, and what the practical differences between them mean for model performance.

Ready to test your understanding of the vocabulary problem? Take this quick quiz to reinforce what you've learned about why word-level tokenization breaks down and why subword approaches are needed.