FastText › The Vocabulary Problem in Word2Vec
Language AI HandbookAll chapters FastText › The Vocabulary Problem in Word2VecOne section · Sections in this chapter2 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before building FastText's solution, it is worth understanding the problem precisely. Word2Vec maintains a vocabulary of words observed during training. Each vocabulary word is represented by a unique embedding vector, and the model learns by predicting which words appear near each other in text. This works well when your test data closely resembles your training data and when your vocabulary is large enough to cover nearly all tokens you will encounter.
Consider what happens inside a Word2Vec model when you present a test sentence containing a word it has not seen.
import numpy as np
# Simulate a Word2Vec vocabulary trained on general English text
word2vec_vocab = {
"run": 0,
"running": 1,
"runner": 2,
"walk": 3,
"walking": 4,
"walker": 5,
"swim": 6,
"happy": 7,
"happily": 8,
}
# Each known word has a 50-dimensional embedding
np.random.seed(42)
word2vec_embeddings = {word: np.random.randn(50) for word in word2vec_vocab}
# Words that appear in real text but are not in vocabulary
problem_words = ["runs", "swam", "swimmer", "runing", "happyness"]Word2Vec vocabulary coverage: ------------------------------------------------------- Vocabulary size: 9 Known words: ['run', 'running', 'runner', 'walk', 'walking', 'walker', 'swim', 'happy', 'happily'] Words that cause failures: 'runs': NOT FOUND (inflected form of 'run') 'swam': NOT FOUND (past tense of 'swim') 'swimmer': NOT FOUND (noun derived from 'swim') 'runing': NOT FOUND (typo for 'running') 'happyness': NOT FOUND (misspelling of 'happiness')
Every word marked "NOT FOUND" will receive no embedding. Downstream models that accept embedding vectors must either skip these words or use a fallback zero vector, both of which lose information. In practice, the fallback zero vector introduces a subtle but persistent bias: every unknown word looks identical to every other unknown word in the embedding space, as if "swam," "runing," and "happyness" were all the same thing.
The problem intensifies across three scenarios that are far more common than toy examples suggest:
-
Morphologically rich languages: Finnish has fifteen grammatical cases; each noun can appear in dozens of forms depending on its grammatical role. Turkish stacks suffixes to express tense, person, plurality, and possession on a single word. A vocabulary-based model would need to memorize tens of thousands of forms per root, which is both impractical and wasteful, because each form is essentially the same concept expressed in a different grammatical context.
-
Domain-specific terminology: Medical literature is dense with terms like "bevacizumab," "electroencephalography," and "hypothyroidism." A model trained on Wikipedia will have none of these, leaving entire sentences without embeddings even though a knowledgeable reader would recognize that "electro" relates to electricity, "encephalo" refers to the brain, and "graphy" denotes measurement. The substructure carries meaning that the atomic word model ignores.
-
Noisy and informal text: Social media writing contains elongated spellings ("soooo"), abbreviations ("gr8"), creative portmanteaux ("hangry"), and deliberate misspellings. These carry clear meaning that a strict vocabulary lookup will miss entirely. A model deployed on social media content will encounter OOV tokens in a substantial fraction of sentences.
FastText addresses all three of these with the same mechanism: decomposing words into character substrings and learning embeddings for those substrings rather than for whole words. The key realization is that morphological information lives at the character level. Prefixes, roots, and suffixes are the meaningful units of word structure, and character n-grams approximate those units without requiring a dedicated morphological analyzer.