Word Tokenization › Why Tokenization Matters
Language AI HandbookAll chapters Word Tokenization › Why Tokenization MattersOne section · Sections in this chapter2 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Every text processing task begins with tokenization. Consider what happens when you search for a word in a document, count word frequencies, or train a language model. All of these require knowing where one word ends and another begins. The decisions you make at this stage set the vocabulary for your entire system, and the vocabulary shapes what patterns the system can and cannot learn.
To understand the stakes, consider a simple word-frequency analysis. If you want to know how often the word "waiting" appears in a text, you need to ensure that "waiting," (with comma) and "waiting." (with period) and "waiting!" (with exclamation mark) all count as instances of "waiting". If your tokenizer leaves punctuation attached, each of these becomes a different vocabulary entry, and your frequency count dramatically underestimates the word's frequency. Multiply this effect across an entire vocabulary and the distortions become severe.
The problem runs deeper than mere counting. Consider training a language model on a corpus where "don't" has never been split. The model learns "don't" as a single token and develops some internal representation for it. But it has learned nothing about the relationship between "don't" and "do not", or between "can't" and "cannot". It treats negation as an opaque property of specific word forms rather than as a compositional structure. A tokenizer that splits contractions exposes this structure, making it much easier for a model to generalize.
A token is a sequence of characters that forms a meaningful unit for processing. In word tokenization, tokens typically correspond to words, punctuation marks, or other linguistically significant elements. The key word is "meaningful": a tokenizer is not just splitting a string; it is making a claim about what units carry information in the language. Different tasks may define "meaningful" differently, which is why there is no single universal tokenizer.
The definition of "word" itself varies by application. For some tasks, "New York" should be a single token because it names a specific place and treating it as two separate words loses that identity. For others, "don't" should split into "do" and "n't" because that reveals the syntactic structure. For bag-of-words document classification, keeping contractions together is fine. For syntactic parsing, splitting them is essential. There's no universal right answer. The best tokenization depends on what you're trying to accomplish.
Tokenization also determines your vocabulary size, and vocabulary size has practical consequences. A vocabulary of 100,000 unique tokens means a model needs 100,000 embedding vectors, one for each token. If you can reduce that vocabulary to 50,000 by better tokenization, you cut the embedding table in half. You also improve coverage: a model that has never seen "colour" (British spelling) might still understand it if "color" and "colour" can both map to the same token after normalization.
# A simple example of why tokenization matters
text = "Dr. Smith's patients don't like waiting. It's 3:30pm!"
# Naive whitespace split
naive_tokens = text.split()
# What we might actually want
expected_tokens = [
"Dr.",
"Smith",
"'s",
"patients",
"do",
"n't",
"like",
"waiting",
".",
"It",
"'s",
"3:30pm",
"!",
]Text: Dr. Smith's patients don't like waiting. It's 3:30pm! Naive whitespace split: ['Dr.', "Smith's", 'patients', "don't", 'like', 'waiting.', "It's", '3:30pm!'] Token count: 8 Linguistically-aware tokenization: ['Dr.', 'Smith', "'s", 'patients', 'do', "n't", 'like', 'waiting', '.', 'It', "'s", '3:30pm', '!'] Token count: 13
The naive approach produces 7 tokens, lumping punctuation with words. A more sophisticated tokenizer produces 13 tokens, separating punctuation and splitting contractions. Which is correct? It depends on your task. For bag-of-words models, you might want "don't" as one token. For syntactic parsing, splitting into "do" and "n't" reveals the underlying structure.
Tokenization also determines your vocabulary. A larger corpus with naive tokenization might produce tokens like "waiting.", "waiting,", and "waiting" as three separate vocabulary entries. Better tokenization consolidates these into a single "waiting" entry plus punctuation tokens, improving vocabulary efficiency. The key insight is that every tokenization decision is also a vocabulary decision: you are implicitly choosing which character sequences count as the same word and which count as different words.
The Zipfian distribution visible in the left plot affects tokenization strategy. Because a tiny number of tokens account for a huge fraction of all occurrences, the decisions you make about high-frequency tokens matter more than decisions about rare ones. The word "the" might appear millions of times; getting its tokenization right matters enormously. An obscure technical term might appear twice in a corpus; whether it is treated as one token or two matters hardly at all.