Subword Tokenization & FastText (2016) › The Solution

FastText addressed these limitations by learning embeddings for character n-grams in addition to word-level embeddings. Instead of treating words as atomic units, FastText represented each word as the sum of embeddings for its character n-grams. This approach allowed the model to construct meaningful representations for unseen words by combining embeddings of their subword components, even if the complete word had never appeared during training.

FastText represented words using a fixed-size set of character n-grams. For a word like "where", with $n=3$, the method would extract character sequences of length 3: "<wh", "whe", "her", "ere", "re>", where "<" and ">" represented word boundaries. The embedding for "where" would be the sum of embeddings for these n-grams, plus optionally a special embedding for the complete word if it appeared in the vocabulary. This representation meant that words sharing character sequences would have related embeddings, even if the complete words were different.

The character n-gram approach captured several types of linguistic information. Prefixes and suffixes, common in morphological systems, appeared as repeated n-grams across words. Words sharing morphological affixes would develop similar embeddings in the relevant components. Root morphemes, appearing in multiple word forms, contributed consistent embedding components. Character-level patterns that indicated semantic or syntactic categories could also emerge, where certain n-gram sequences were associated with specific linguistic functions.

The method supported variable n-gram sizes, typically using n-grams from 3 to 6 characters in length. Longer n-grams captured more word-specific information, while shorter n-grams captured more general morphological patterns. The combination of multiple n-gram sizes allowed the model to balance specificity and generalizability. Very short n-grams could appear across many words, learning general patterns, while longer n-grams provided word-specific information when available.

FastText used the same Skip-Gram architecture as word2vec, with one modification: it computed a word's embedding as the sum of its n-gram embeddings. During training, the model learned both word-level embeddings (for frequent words) and n-gram embeddings (for character sequences). The prediction task remained the same: given a target word, predict its context words. However, when computing the embedding for the target word, FastText summed the embeddings of all its character n-grams.

For example, to compute the embedding for "running", FastText would extract n-grams like "<ru", "run", "unn", "nni", "nin", "ing", "ng>", and potentially others depending on the n-gram size range. The word embedding would be the sum of embeddings for these n-grams, possibly plus a word-specific embedding if "running" appeared frequently enough to warrant its own representation. This summed representation could then be used in the Skip-Gram prediction task, where the model learned to predict context words given the summed n-gram embeddings.

The training process used the same negative sampling approach as word2vec, maintaining computational efficiency while learning from large corpora. However, the parameters being learned included word embeddings and n-gram embeddings that could be shared across many words. This parameter sharing meant that frequent n-grams, appearing in many words, received more training signal and developed more reliable embeddings. Rare words benefited from these well-trained n-gram embeddings, even if the words themselves appeared infrequently.

The key advantage of FastText emerged when processing words not seen during training. For an unseen word, FastText could still construct an embedding by extracting its character n-grams and summing their embeddings. If the n-grams had been seen in other words during training, their embeddings would provide meaningful information about the unseen word's likely meaning. For example, if "unhappiness" was unseen but "unhappy" and "happiness" had appeared during training, the n-grams "<un", "hap", "piness", "ess>" might have learned embeddings that could combine to represent "unhappiness" reasonably well.

This capability was particularly valuable for morphologically rich languages, where unseen words often shared significant subword structure with seen words. A new inflected form of a known root word could be represented through its shared morphological components. Even completely novel words could benefit if their character n-grams matched patterns learned from similar words. The method gracefully degraded: words with completely novel character patterns would still receive embeddings, but these embeddings might be less informative until the model encountered similar patterns in training.