The Vocabulary Problem › The Case for Subword Units
Language AI HandbookAll chapters The Vocabulary Problem › The Case for Subword UnitsOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
All the challenges we've examined point toward a common solution: stop treating words as indivisible atoms. If we break words into smaller, reusable pieces, we gain several important properties simultaneously. This unification is what makes subword tokenization so powerful: it doesn't just solve one problem, it dissolves the entire cluster of problems that plagued word-level tokenization.
Think of subword units as the natural vocabulary of morphology. Human languages are built on the principle of double articulation: a finite set of meaningless sounds (phonemes) combines to form a finite set of meaningful minimal units (morphemes), which combine to form an unbounded set of words and sentences. Subword tokenization approximates this structure without requiring linguistic expertise: it learns, from data, which character sequences are worth treating as units.
The benefits of moving from words to subwords are:
-
No OOV: Any word, however novel, can be decomposed into known subword pieces. At worst, each character is its own token. The OOV problem disappears structurally, not because the vocabulary is large enough to cover everything, but because the decomposition can always fall back to smaller pieces.
-
Parameter sharing: The piece "un-" learned from "undo" transfers to "unfair," "unhappy," "unclear," and any future "un-" word. Every time the model processes a word containing a known subword, it draws on everything it has learned from all other contexts where that subword appeared. This is the key mechanism: subword representations generalize across words in a way that whole-word representations cannot.
-
Manageable vocabulary: Thousands of subword units can generate millions of word forms. A vocabulary of 32,000 pieces can represent essentially all of English, most other European languages, and a substantial fraction of any language written in Latin script, without any OOV tokens and without the memory burden of millions of word entries.
-
Morphological decomposition: Complex words naturally break at morphological boundaries. "Preprocessing" splits into "pre," "process," and "ing," three meaningful units whose combination correctly suggests the meaning. This decomposition is learned from data, not encoded by hand, but it often aligns surprisingly well with linguistic morpheme boundaries.
-
Cross-lingual transfer: Because subword methods learn language-independent character patterns, a multilingual model can share vocabulary across languages that use the same script. Many words in related languages share prefixes, roots, or suffixes, and subword models can exploit this structure without any explicit cross-lingual supervision.
from collections import Counter
# Demonstration of subword decomposition
subword_decompositions = {
"unhappiness": ["un", "happi", "ness"],
"unbelievable": ["un", "believ", "able"],
"transformers": ["transform", "ers"],
"preprocessing": ["pre", "process", "ing"],
"internationalization": ["inter", "national", "ization"],
"ChatGPT": ["Chat", "G", "PT"],
"COVID19": ["CO", "VID", "19"],
}
# Measure how often subword pieces are reused across words
all_pieces = []
for decomposition in subword_decompositions.values():
all_pieces.extend(decomposition)
piece_counts = Counter(all_pieces)Subword Decomposition Examples ================================================================= Word Decomposition ----------------------------------------------------------------- unhappiness "un" + "happi" + "ness" unbelievable "un" + "believ" + "able" transformers "transform" + "ers" preprocessing "pre" + "process" + "ing" internationalization "inter" + "national" + "ization" ChatGPT "Chat" + "G" + "PT" COVID19 "CO" + "VID" + "19" Reused subword pieces (shared across multiple words): 'un' appears in 2 words
The decomposition table illustrates the key property: diverse words share subword pieces. "Unhappiness" and "unbelievable" both contain "un-", so learning that prefix from one word immediately benefits processing the other. "Preprocessing" and any other "-ing" word share that suffix. Novel words like "ChatGPT" and "COVID19" split into segments that, while not morphologically meaningful in the traditional sense, are learnable patterns the model can encode. The model might learn that "GPT" tends to appear in contexts related to language models, so any word containing "GPT" inherits that context. Every piece learned from one word is immediately available for composing any other word that contains it.
This approach avoids [UNK] and shares parameters through subword reuse. Parameter sharing through subword reuse means that the model's effective training data for any given morpheme is the sum of its occurrences across all words rather than occurrences of any particular word form. "Un-" appears in "undo," "unfair," "unhappy," "unreal," "unbelievable," and hundreds of other words. A subword model trained on all of these forms builds a rich representation of what "un-" means, and applies that representation when it encounters a new "un-" word. This is fundamentally different from word-level models, where "undo" and "unfair" are entirely separate entries with no shared parameters.