Subword Tokenization & FastText (2016) › The Problem

Word-level embedding methods like word2vec treated each word as an indivisible unit, assigning a single embedding vector to each vocabulary item. This approach worked well for frequent words that appeared many times in training corpora, allowing the model to learn reliable representations. However, it failed catastrophically for rare words and out-of-vocabulary terms. A word appearing only a handful of times during training received insufficient updates, leading to embeddings that captured little meaningful information. Completely unseen words received no representation at all, forcing systems to fall back to random vectors or special unknown word tokens that provided no semantic information.

The problem became particularly acute in morphologically rich languages. In languages like Finnish, a single root word might generate dozens of inflected forms through compounding and suffixation. The word "talo" (house) might appear as "talossa" (in the house), "talosta" (from the house), "taloon" (into the house), and many other variants. Each inflection represented a distinct vocabulary item in word-level models, fragmenting the available training data across many related forms. Since each form appeared less frequently, the embeddings learned for these variants were less reliable. More fundamentally, word-level embeddings could not capture the systematic relationships between these morphologically related forms, missing the linguistic insight that they shared a common root.

Even in morphologically simpler languages like English, rare words posed challenges. Domain-specific terminology appeared infrequently in general-purpose training corpora. Medical terms, scientific concepts, technical jargon, and proper nouns often had rich semantic content but sparse occurrence patterns. A text classification system trained on news articles might encounter specialized vocabulary when processing scientific papers, patient records, or legal documents. Without representations for these terms, the system could not use the semantic information they conveyed.

Newly coined words and evolving vocabulary presented another limitation. Language changes constantly, with new words entering common usage through technological change, social media, cultural shifts, and domain-specific developments. A word embedding model trained on a static corpus could not represent words that emerged after training. This limitation particularly affected user-generated content, where creative spelling appeared alongside slang and shortened or newly coined forms. Social media text exposed the brittleness of word-level approaches as its informal vocabulary changed rapidly.

The computational implications of large vocabularies also created practical problems. Word-level models required storing and updating embeddings for every unique word in the vocabulary. For large-scale systems processing multiple languages or domains, vocabularies could contain millions of unique words. This vocabulary explosion increased memory requirements and computational costs during training and inference. The softmax normalization over large vocabularies, even with negative sampling approximations, remained computationally expensive. Large vocabularies also increased the risk of overfitting, where models memorized word-specific patterns rather than learning generalizable linguistic relationships.

Languages with rich morphological systems compounded these vocabulary problems. Agglutinative languages, where words can have many morphemes concatenated together, generated enormous vocabularies where most words appeared only once or twice. This data sparsity made it difficult to learn reliable embeddings. The atomic treatment of words also missed linguistic generalizations: morphologically related words should share similar embeddings because they have common roots and systematic morphological processes. Word-level embeddings failed to capture these relationships explicitly, relying instead on co-occurrence patterns that might be sparse for related but infrequent word forms.