Subword Tokenization & FastText (2016) › 2016: Subword Tokenization and FastText

By 2016, word embeddings had become a fundamental tool in natural language processing, with word2vec demonstrating that dense vector representations could capture rich semantic and syntactic relationships. However, the field confronted a persistent challenge: how to handle words that appeared rarely or never in training corpora. These out-of-vocabulary words presented a fundamental limitation for embedding methods that treated each word as an atomic unit. Facebook AI Research, led by Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov, introduced FastText, a method that addressed this limitation by learning embeddings for words and for subword components. This innovation opened new possibilities for handling morphologically rich languages, rare words, and domain-specific terminology.

The problem of rare and unseen words had been recognized since the earliest embedding methods. Word2vec and similar approaches assigned each word in the vocabulary a unique embedding vector. When encountering a word that wasn't in the training vocabulary, these methods had no principled way to represent it. This limitation particularly affected morphologically rich languages like Finnish, Turkish, or German, where words could have dozens of inflected forms, each appearing infrequently in corpora. Even in English, technical terminology, proper nouns, and new coinages presented challenges. A sentiment analysis system trained on general text might encounter domain-specific terms in user reviews, medical terms in patient reports, or technical jargon in engineering documents, all completely unrepresented in its vocabulary.

FastText emerged at a moment when the NLP community was seeking solutions to these vocabulary limitations. The method built on insights from morphological analysis and character-level language modeling, which had shown that subword information could be valuable for understanding word structure and meaning. By learning embeddings for character n-grams alongside word-level embeddings, FastText could construct representations for unseen words by combining the embeddings of their subword components. A word like "unseen" could be represented through embeddings for its character sequences, such as "<un", "uns", "nse", "see", "een", "en>", where "<" and ">" indicated word boundaries.

FastText showed that morphological and subword information improved embeddings for rare and unseen words. This result influenced later tokenization methods, including Byte Pair Encoding (BPE) and SentencePiece, that became standard in transformer-based language models. FastText also showed that a model could represent a word through its internal structure instead of treating it as an atomic unit.