Word Tokenization › Subword Tokenization Preview
Language AI HandbookAll chapters Word Tokenization › Subword Tokenization PreviewOne section · Sections in this chapter12 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Modern neural NLP systems often use subword tokenization instead of word tokenization. Methods like Byte Pair Encoding (BPE), WordPiece, and SentencePiece break words into smaller units that are discovered from the training data itself, rather than defined by hand-crafted rules. Understanding why this shift happened requires understanding the limitations of word tokenization.
The fundamental problem with word tokenization for neural networks is the vocabulary problem. A large English corpus might contain hundreds of thousands of unique word forms. Each unique word needs an embedding vector. Words that appear rarely in training receive poor embeddings because the model has few examples to learn from. Words that never appear in training are completely out-of-vocabulary (OOV) and cannot be handled at all. This is particularly problematic for morphologically rich languages (like Finnish or Turkish) where a single root word might have dozens of inflected forms, each requiring its own embedding.
Subword tokenization solves this by decomposing words into smaller units. Rather than having a separate embedding for "running", "runner", "run", and "runs", a subword tokenizer might represent all of these using a combination of "run" and suffixes like "-ning", "-ner", and "-s". The model learns representations for these subword units, and new words can be handled by decomposing them into known subwords.
# Preview of subword tokenization concepts
word = "unhappiness"
# Possible subword decompositions
decompositions = {
"Character": list(word),
"Morpheme": ["un", "happi", "ness"],
"BPE-style": ["un", "happ", "iness"],
"Word": [word],
}Word: 'unhappiness' Different granularities: Character : ['u', 'n', 'h', 'a', 'p', 'p', 'i', 'n', 'e', 's', 's'] (11 tokens) Morpheme : ['un', 'happi', 'ness'] (3 tokens) BPE-style : ['un', 'happ', 'iness'] (3 tokens) Word : ['unhappiness'] (1 tokens)
Subword tokenization offers advantages: smaller vocabularies, better handling of rare words, and no out-of-vocabulary problem. We'll explore these methods in detail in a later chapter.
The following table illustrates the granularity spectrum for the word "unhappiness":
| Granularity | Tokens | Token Count | Vocab Implication |
|---|---|---|---|
| Word | ["unhappiness"] | 1 | Large vocabulary needed |
| Morpheme | ["un", "happi", "ness"] | 3 | Meaningful subunits |
| BPE-style | ["un", "happ", "iness"] | 3 | Data-driven splits |
| Character | ["u","n","h","a","p","p","i","n","e","s","s"] | 11 | Fixed small vocabulary |
Finer granularities (character-level) produce more tokens but require smaller vocabularies since every word can be built from a small set of characters. Coarser granularities (word-level) produce fewer tokens but require larger vocabularies to cover all possible words. Subword methods like BPE and WordPiece occupy a middle ground, balancing vocabulary size with sequence length. The key insight is that this tradeoff is not fixed: by choosing the vocabulary size as a hyperparameter, subword methods let you control where on this spectrum you want to be.