FastText › Summary

FastText extends Word2Vec by replacing word-level atomic representations with sums of character n-gram embeddings. This single architectural change yields three practical benefits: handling of out-of-vocabulary words, morphological awareness through shared n-gram structure, and less brittleness to typos and noisy text. The training objective remains skip-gram with negative sampling, but gradients now flow back to all n-gram embeddings in a center word, allowing shared learning across morphological variants. The hashing trick bounds memory usage at a fixed bucket count regardless of vocabulary size. This makes deployment practical at scale.

Key takeaways from this chapter:

• Subword representation: Each word is decomposed into character n-grams (default lengths 3 to 6), with boundary markers separating positional from interior contexts
• Embedding computation: A word's embedding is the sum of its n-gram embeddings, plus an optional word-level term for known vocabulary words
• OOV handling: Any word can receive an embedding by summing whatever n-gram embeddings were learned, enabling zero-shot generalization to unseen words, typos, and morphological variants
• Morphological awareness: Words sharing the same root share n-grams, so morphological variants naturally cluster in embedding space even for words never seen during training
• Hash bucketing: N-grams are hashed to a fixed number of buckets to control memory usage, trading minor accuracy for large memory savings at production scale
• Language applicability: FastText excels on morphologically rich alphabetic languages but has limitations for logographic scripts like Chinese
• Context independence: Like all static embeddings, FastText cannot distinguish word senses from context, which limits its ability to represent polysemous words accurately

The next chapter on embedding evaluation will give us tools to measure these properties quantitatively: how to test similarity, how to assess OOV coverage, and how to detect when embedding quality is insufficient for a downstream task. The chapter after that introduces contextual embeddings with ELMo, which addresses the context-independence limitation at the cost of higher computational complexity.