Subword Tokenization & FastText (2016) › Limitations

Despite its advances, FastText retained limitations from the word-level embedding paradigm while introducing new challenges. The method still produced static embeddings: each word (or its n-gram composition) received a single representation regardless of context. FastText could not capture polysemy or context-dependent meaning variations. The word "bank" would have a fixed representation whether it appeared in financial or geographical contexts, though the character n-grams might provide some morphological consistency.

The character n-gram approach, while helpful for handling rare words, introduced ambiguity when n-grams were shared across unrelated words. Words with similar character sequences but different meanings might develop overly similar embeddings due to shared n-grams. The method assumed that morphological similarity correlated with semantic similarity, which held for many cases but failed for others. False morphological relationships could emerge from shared character patterns with no linguistic significance.

Computational overhead increased compared to word2vec, as FastText needed to extract n-grams, sum embeddings, and potentially handle larger parameter spaces for n-gram embeddings. While still efficient, the additional computation made FastText somewhat slower than pure word-level approaches. The n-gram vocabulary could also be large, requiring significant memory for storing n-gram embeddings, though this was typically smaller than the word vocabulary explosion it helped mitigate.

The method's effectiveness depended on the quality and diversity of training data. Rare n-grams that appeared only in specific domains might not develop meaningful embeddings if those domains were underrepresented in training corpora. FastText worked best when character patterns in unseen words matched patterns learned from similar words, but completely novel morphological structures might still struggle.

FastText embeddings also captured the same statistical associations as word2vec, rather than true semantic understanding. Words that co-occurred frequently developed similar embeddings, even when this co-occurrence reflected topical rather than semantic relationships. The subword approach helped with rare words but didn't fundamentally address the distributional semantics limitations of word embedding methods.

The fixed n-gram size range meant FastText might miss important morphological boundaries. In some languages, morpheme boundaries don't align cleanly with character boundaries, and fixed-length n-grams might not capture these linguistic structures optimally. Languages with complex morphological systems might require morphological analysis to properly segment words, which FastText didn't explicitly perform.