The Vocabulary Problem › The Curse of Rare Words
Language AI HandbookAll chapters The Vocabulary Problem › The Curse of Rare WordsOne section · Sections in this chapter4 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Even when rare words make it into the vocabulary, they face a different and subtler problem: they simply cannot be learned well. This is perhaps the most insidious aspect of word-level tokenization because it creates a two-tier system where frequent words have rich, reliable representations and rare words have poor, unreliable ones, and the boundary between them is blurry.
To understand why this happens, recall how word embeddings are learned. Each word's vector representation is updated based on the contexts in which it appears. Each time a word occurs in training data, its embedding receives a gradient signal telling it to be more similar to its context words. A word appearing 100,000 times receives 100,000 such updates across training, each gradually refining the embedding to better capture the word's semantic neighborhood. A word appearing 10 times receives only 10 gradient updates, and its embedding remains close to its random initialization.
The problem comes from both the quantity of updates and their diversity. A frequent word is observed in many different contexts: formal and informal sentences, different topics, different syntactic positions. This diversity of contexts gives the model a varied signal about the word's meaning. A rare word might appear in only one or two semantic contexts, so even if it appears a dozen times, those appearances may all be similar. This provides a narrow and potentially unrepresentative signal.
Poorly Learned Representations
The result is that rare word embeddings are poorly learned. They may be nearly random vectors, barely moved from their random initialization. This isn't a limitation of the learning algorithm: it is a fundamental consequence of having too little signal. No learning algorithm can extract information that isn't present in the data, and a word that appears 5 times simply doesn't provide enough context to reliably determine its meaning relative to all other words.
Think of learning word meanings the way a child learns vocabulary: through repeated exposure in varied contexts. You learn what "serendipity" means because you've heard it used to describe unexpected good fortune across dozens of situations: stories, movies, casual conversation. If you heard the word only once, in a single sentence, your understanding would be vague and unreliable. Word embedding training faces the exact same constraint, just formalized mathematically.
This creates a vicious cycle. Rare words have poor embeddings, which means they contribute little reliable signal to downstream tasks, which means task-specific fine-tuning can't improve them because there isn't enough data. The model learns to ignore rare words or treat them as noise, which reinforces their poor representation. Even if you include a rare word in the vocabulary, you often get little practical benefit over [UNK] because the embedding is so poorly initialized that the model can't make good use of it.
The Minimum Frequency Cutoff
To avoid polluting the vocabulary with poorly-learned embeddings, most word embedding methods impose a minimum frequency cutoff. Words appearing fewer than, say, 5 times are excluded from the vocabulary entirely and mapped to [UNK]. The rationale is that a word seen only once or twice doesn't have enough signal to train a useful embedding, so it's better to exclude it and keep the model clean than to include it with a near-random vector that might actively mislead downstream tasks.
Word2Vec's min_count parameter defaults to 5 for this reason. GloVe's co-occurrence matrix construction ignores words below a frequency threshold. FastText, which we'll discuss shortly, is one of the few word-level methods that partially addresses this problem by using character n-grams. But for standard word embedding methods, the minimum frequency cutoff is a necessary compromise.
# Simulated word frequency distribution across a corpus
word_frequencies = {
"the": 1_000_000,
"and": 500_000,
"learning": 50_000,
"neural": 25_000,
"transformer": 5_000,
"bert": 2_000,
"roberta": 500,
"deberta": 100,
"electra": 50,
"xlnet": 20,
"reformer": 5,
"longformer": 3,
"linformer": 1,
}
# Apply different frequency cutoffs and measure the effect
cutoffs = [1, 5, 20, 100]
vocab_sizes_by_cutoff = {}
for cutoff in cutoffs:
vocab_sizes_by_cutoff[cutoff] = sum(
1 for freq in word_frequencies.values() if freq >= cutoff
)Effect of Minimum Frequency Cutoff ======================================================= Cutoff Vocab Size Words Excluded ------------------------------------------------------- min_count = 1 13 0 min_count = 5 11 2 min_count = 20 10 3 min_count = 100 8 5
The table makes the tradeoff concrete. Setting min_count=1 keeps all 13 words, but entries like "linformer" (appearing only once) will have nearly random embeddings that add noise rather than signal. Raising the cutoff to min_count=5 excludes only the three rarest words, "linformer," "longformer," and "reformer," while improving embedding quality for everything that remains. At min_count=100, only 8 words survive the cutoff. In practice, cutoffs of 5 to 10 are common across major toolkits, but every excluded word becomes OOV at inference time, replaced by [UNK] with no semantic content.
The key insight is that the frequency cutoff is not a solution to the rare word problem: it is an admission of defeat. You're not fixing the problem; you're acknowledging that word-level representations can't handle rare words and choosing to discard them rather than represent them badly. This is a perfectly reasonable engineering decision given the constraints, but it means that rare words, which include many of the most semantically specific and important terms in technical text, simply don't exist for word-level models.