Subword Tokenization & FastText (2016) › Applications and Impact

FastText quickly found applications across diverse NLP tasks where handling rare words and domain-specific vocabulary was important. In text classification, FastText embeddings improved performance on tasks involving technical domains, morphologically rich languages, and social media text. The method's ability to represent unseen words helped classifiers maintain performance when processing text from domains or genres not well-represented in training data.

Named entity recognition systems benefited from FastText's handling of proper nouns and technical terms. Many named entities appear infrequently in training corpora, making word-level embeddings unreliable. FastText could represent these entities through their character n-grams and use morphological patterns even for rare proper nouns. In multilingual text, shared character patterns also helped represent words from languages absent from the training vocabulary.

Information retrieval applications used FastText embeddings to improve matching between queries and documents containing rare or technical vocabulary. The subword representations helped capture semantic similarity even when exact word matches were absent. A query containing "cardiovascular" could match documents mentioning "cardiac" or "vascular" through shared character n-grams, even if "cardiovascular" itself was rare.

FastText also demonstrated effectiveness on morphologically rich languages. In languages like Finnish, Czech, or Arabic, where words have many inflected forms, FastText embeddings captured systematic relationships between related words. Their shared n-gram components caused related forms to cluster in the embedding space, reflecting their morphology. This made FastText useful for multilingual applications spanning different morphological systems.

The method's computational efficiency, similar to word2vec, made it practical for large-scale applications. FastText could train on corpora with billions of words using standard hardware, and inference remained fast even with n-gram extraction and summation. Pre-trained embeddings for many languages also made representations that handled rare forms readily available.

Social media and user-generated content applications used FastText to handle informal and rapidly changing language. Character n-grams could represent misspellings as well as shortened or newly coined forms, even when a form was novel. This made FastText useful for sentiment and content-moderation tasks, including spam detection.