FastText › Handling Out-of-Vocabulary Words
Language AI HandbookAll chapters FastText › Handling Out-of-Vocabulary WordsOne section · Sections in this chapter7 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The OOV case is where FastText's design pays off most dramatically. Let's test words the model never saw during training. For an OOV word, the model computes its embedding by extracting n-grams, looking up any that appear in the n-gram table (which was built from training vocabulary words), and summing their embeddings. N-grams not in the table are simply skipped.
oov_words = ["runs", "walked", "swimmers", "jumped", "runing"]
in_vocab_set = set(vocab)
oov_results = {}
for word in oov_words:
emb = model.get_embedding(word)
sim_words = model.most_similar(word, top_n=3)
oov_results[word] = {
"in_vocab": word in in_vocab_set,
"emb_norm": float(norm(emb)),
"similar": sim_words,
}Out-of-vocabulary word handling:
-------------------------------------------------------
'runs' (in vocab):
Embedding norm: 7.8330
Most similar: [('run', 0.852), ('runner', 0.69), ('runners', 0.576)]
'walked' (OOV):
Embedding norm: 4.5201
Most similar: [('walks', 0.797), ('walker', 0.775), ('walk', 0.695)]
'swimmers' (in vocab):
Embedding norm: 11.4488
Most similar: [('day', 0.816), ('every', 0.57), ('train', 0.49)]
'jumped' (OOV):
Embedding norm: 4.4676
Most similar: [('jumps', 0.871), ('jumper', 0.79), ('jump', 0.768)]
'runing' (OOV):
Embedding norm: 5.5047
Most similar: [('running', 0.906), ('run', 0.864), ('the', 0.694)]OOV words receive meaningful, non-trivial embeddings and find their morphological relatives through nearest-neighbor search. "Walked" (past tense, absent from training) correctly identifies "walk," "walking," and "walker" as its nearest neighbors because the 3-gram "wal" and 4-gram "walk" appear in all of them. "Runing" (a typo for "running") finds "running" at the top because they differ by only one character and share nearly all n-grams.
This robustness to typos degrades gracefully with edit distance. A single-character change preserves most n-grams; two changes preserve fewer; severe distortions eventually produce embeddings only weakly related to the intended word. The degradation is smooth rather than catastrophic, which means FastText handles real-world input noise far more gracefully than word-level models, which fail completely on any deviation from known forms.
An important subtlety: the quality of the OOV embedding depends on how many of its n-grams were seen during training. A word that shares n-grams with many training words will get a rich embedding. A word from an entirely different language or script may produce an embedding of near-zero norm if none of its n-grams appear in the table. This is a meaningful signal: a low-norm OOV embedding indicates a word that the model has no basis for representing.
