FastText › Limitations and What Comes Next

FastText advances Word2Vec, but it inherits one critical limitation from the static embedding paradigm and introduces one new concern of its own.

Every word has exactly one embedding regardless of which sentence it appears in. The word "bank" receives the same vector whether you write "the river bank" or "the bank account." FastText has no mechanism to distinguish these senses because the embedding is computed from character n-grams alone, with no awareness of the surrounding text. This is not a flaw in FastText specifically but a property of all static embedding methods: the representation is computed once per word, offline, before any inference happens.

The problem is real and consequential. Polysemous words are extremely common in English and other natural languages. WordNet catalogs an average of 2.6 senses per noun. For words in the long tail of sense distributions, a single averaged vector is a poor representation of any individual sense. Downstream models that use static embeddings must implicitly recover the correct sense from context, which places an extra burden on the task-specific model and limits how much of the semantic signal the embeddings can provide.

The long-term solution to context independence came from a different direction. ELMo, introduced in 2018, used bidirectional LSTMs to produce word representations that depend on the entire sentence context. The same word in different sentences produces different vectors. BERT extended this with the transformer architecture, producing context-sensitive representations that captured word senses, syntactic roles, coreference, and other discourse-level phenomena. FastText embeddings and BERT contextual embeddings are designed for different situations: FastText is fast, memory-efficient, and well-suited to morphologically rich languages, while BERT produces richer representations at the cost of much higher compute and memory requirements.

The n-gram mechanism cannot distinguish meaningful morphological segments from arbitrary character sequences. "xyzqwk" produces a non-zero embedding by summing its n-gram vectors, even though no human would recognize it as a word from any known language. Applications that rely on zero-norm vectors to signal unknown tokens must add explicit out-of-vocabulary detection logic. In practice this is an inconvenience rather than a fundamental problem, but it matters when deploying FastText in a pipeline that needs to flag anomalous inputs.

Related to this: FastText embeddings for very short words (two or three characters) can be low quality because short words have few n-grams. The word "a" with the default minimum of 3 and maximum of 6 produces just the single n-gram "<a>," which gives it almost no representational capacity. Short function words in general tend to be handled less well by FastText than by word-level models, which is the reverse of the situation for long morphologically complex words.

Even with hashing, storing 2 million 300-dimensional float vectors requires about 2.4 GB for the n-gram table alone. The full binary models released by Meta can exceed 7 GB when word-level embeddings are included. This makes cold-start deployment on mobile devices or in memory-constrained environments challenging. Techniques like product quantization (compressing each embedding vector into a small code) and model distillation (training a smaller model to mimic the larger one) are commonly used to reduce the footprint.

The 300-dimensional default also means that every embedding operation involves multiplying or adding 300-dimensional vectors, which is fast on modern hardware but adds up when processing millions of documents in a real-time pipeline. For latency-sensitive applications, smaller dimensionality (50 or 100 dimensions) often provides a good trade-off between speed and accuracy.

Character n-grams assume that subword character sequences carry meaning. This assumption is sound for alphabetic and syllabic writing systems, where characters map to sounds and sounds map to meaning. But for Chinese, Japanese kanji, and other logographic systems, individual characters often map directly to morphemes or entire words. The n-gram of two adjacent characters may be entirely meaningless as a linguistic unit, or it may be a compound word, with no predictable relationship between character position and meaning. Chinese NLP typically uses word-level tokenization (with dedicated Chinese word segmentation tools) or character-level models rather than FastText's fixed-length n-gram approach. Specialized methods exist for these scripts, and FastText's published results for Chinese and Japanese are generally weaker than for alphabetic languages.

Despite these limitations, FastText occupies an important and durable niche. For downstream tasks on morphologically rich languages, for applications where OOV robustness matters, and for deployment scenarios where a compact static embedding model is preferred over a large contextual model, FastText consistently outperforms Word2Vec with modest additional computational cost. Pretrained FastText models for 157 languages are freely available from Meta, making it easy to use without training from scratch.