FastText › FastText: Subword Embeddings and Character N-grams

Word2Vec opened the door to dense word representations learned from context. But it contains a subtle assumption that becomes a serious limitation at scale: every word is an atom. The embedding for "running" is entirely unrelated to the embedding for "run." They might live nearby in vector space after training, but that proximity is coincidence, not design. The model had to observe each word independently and hope its training signal was strong enough to push related words near each other.

This atomic view has concrete consequences. Any word absent from the training corpus receives no embedding at all. For English, this means rare words, proper nouns, typos, and new coinages all fail silently. The model simply has no answer for them, and downstream systems must cope with the gap through fallbacks that lose information. For morphologically rich languages like Finnish, Turkish, or German, the situation is far worse. Finnish nouns can take over a thousand different inflected forms. German constructs compound words without spaces, producing tokens like "Lebensversicherungsgesellschaft" (life insurance company) that almost never appear twice in any corpus. A Word2Vec model trained on German text cannot possibly memorize every compound, so most compounds are perpetually out-of-vocabulary.

FastText, introduced by Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov at Facebook AI Research in 2017, solves this by abandoning the atomic view. Instead of treating a word as a single unit, FastText treats it as a collection of overlapping character substrings called n-grams. "Running" becomes a bag containing the substrings "run," "unn," "nni," "nin," "ing," and several longer fragments. The word's final embedding is the sum of its n-gram embeddings. This one change, replacing a single lookup with a sum over substrings, is what makes FastText capable of constructing embeddings for words it has never seen.

The insight is borrowed from classical computational linguistics. For decades, researchers used character n-grams to represent documents for language identification and spam filtering, precisely because character substrings are robust to morphological variation and noise. FastText brings that robustness into the neural embedding framework that Word2Vec established.

This chapter develops FastText from first principles. We begin with the vocabulary problem in full detail, then work through the n-gram decomposition, the mathematical training objective, the hashing trick for memory efficiency, and finally examine where FastText still falls short and what came next. By the end, you will understand both why subword embeddings are so powerful and where the next generation of models had to go.