FastText: Subword Embeddings, Character N-grams

Word2Vec opened the door to dense word representations learned from context. But it contains a subtle assumption that becomes a serious limitation at scale: every word is an atom. The embedding for "running" is entirely unrelated to the embedding for "run." They might live nearby in vector space after training, but that proximity is coincidence, not design. The model had to observe each word independently and hope its training signal was strong enough to push related words near each other.

This atomic view has concrete consequences. Any word absent from the training corpus receives no embedding at all. For English, this means rare words, proper nouns, typos, and new coinages all fail silently. The model simply has no answer for them, and downstream systems must cope with the gap through fallbacks that lose information. For morphologically rich languages like Finnish, Turkish, or German, the situation is far worse. Finnish nouns can take over a thousand different inflected forms. German constructs compound words without spaces, producing tokens like "Lebensversicherungsgesellschaft" (life insurance company) that almost never appear twice in any corpus. A Word2Vec model trained on German text cannot possibly memorize every compound, so most compounds are perpetually out-of-vocabulary.

FastText, introduced by Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov at Facebook AI Research in 2017, solves this by abandoning the atomic view. Instead of treating a word as a single unit, FastText treats it as a collection of overlapping character substrings called n-grams. "Running" becomes a bag containing the substrings "run," "unn," "nni," "nin," "ing," and several longer fragments. The word's final embedding is the sum of its n-gram embeddings. This one change, replacing a single lookup with a sum over substrings, is what makes FastText capable of constructing embeddings for words it has never seen.

The insight is borrowed from classical computational linguistics. For decades, researchers used character n-grams to represent documents for language identification and spam filtering, precisely because character substrings are robust to morphological variation and noise. FastText brings that robustness into the neural embedding framework that Word2Vec established.

This chapter develops FastText from first principles. We begin with the vocabulary problem in full detail, then work through the n-gram decomposition, the mathematical training objective, the hashing trick for memory efficiency, and finally examine where FastText still falls short and what came next. By the end, you will understand both why subword embeddings are so powerful and where the next generation of models had to go.

Before building FastText's solution, it is worth understanding the problem precisely. Word2Vec maintains a vocabulary of words observed during training. Each vocabulary word is represented by a unique embedding vector, and the model learns by predicting which words appear near each other in text. This works well when your test data closely resembles your training data and when your vocabulary is large enough to cover nearly all tokens you will encounter.

Consider what happens inside a Word2Vec model when you present a test sentence containing a word it has not seen.

Every word marked "NOT FOUND" will receive no embedding. Downstream models that accept embedding vectors must either skip these words or use a fallback zero vector, both of which lose information. In practice, the fallback zero vector introduces a subtle but persistent bias: every unknown word looks identical to every other unknown word in the embedding space, as if "swam," "runing," and "happyness" were all the same thing.

The problem intensifies across three scenarios that are far more common than toy examples suggest:

• Morphologically rich languages: Finnish has fifteen grammatical cases; each noun can appear in dozens of forms depending on its grammatical role. Turkish stacks suffixes to express tense, person, plurality, and possession on a single word. A vocabulary-based model would need to memorize tens of thousands of forms per root, which is both impractical and wasteful, because each form is essentially the same concept expressed in a different grammatical context.

• Domain-specific terminology: Medical literature is dense with terms like "bevacizumab," "electroencephalography," and "hypothyroidism." A model trained on Wikipedia will have none of these, leaving entire sentences without embeddings even though a knowledgeable reader would recognize that "electro" relates to electricity, "encephalo" refers to the brain, and "graphy" denotes measurement. The substructure carries meaning that the atomic word model ignores.

• Noisy and informal text: Social media writing contains elongated spellings ("soooo"), abbreviations ("gr8"), creative portmanteaux ("hangry"), and deliberate misspellings. These carry clear meaning that a strict vocabulary lookup will miss entirely. A model deployed on social media content will encounter OOV tokens in a substantial fraction of sentences.

FastText addresses all three of these with the same mechanism: decomposing words into character substrings and learning embeddings for those substrings rather than for whole words. The key realization is that morphological information lives at the character level. Prefixes, roots, and suffixes are the meaningful units of word structure, and character n-grams approximate those units without requiring a dedicated morphological analyzer.

The core data structure in FastText is the character n-gram. For any word, FastText first wraps it in boundary markers, then extracts every contiguous substring of characters within a configured length range. The result is a set of overlapping fragments that collectively encode the word's character structure.

The function below extracts all n-grams within a length range. This is the literal preprocessing step applied to every word in the training corpus:

The word "where" produces 18 n-grams spanning lengths 3 to 6. Notice what these substrings encode. The 3-gram "<wh" can only appear at the very start of a word: the "<" marker ensures this. Similarly, "re>" can only appear at a word ending. The internal 3-gram "her" has no positional constraint, so it can match "hero," "mother," "together," and any other word containing that substring in the middle. This positional disambiguation is one of the key functions of the boundary markers.

The n-gram length range is a deliberate design choice. A minimum of 3 avoids extremely short substrings like "re" or "he" that appear in so many words they carry little discriminative information. A maximum of 6 is long enough to capture most productive English morphemes: the suffix "-ation" is 6 characters, "-ment" is 4, the prefix "pre-" is 3. For languages with longer morphemes, the maximum can be extended.

Boundary markers do more than look like a clever trick. They fundamentally change which words share n-grams and by how much. Without markers, "where," "hero," and "other" would all share the trigram "her," conflating three very different words in ways that do not reflect their linguistic relationship. With markers, each word develops a distinct positional fingerprint.

Without markers, "where," "hero," and "other" would all share the trigram "her," merging their embeddings in a way that does not reflect a linguistic relationship. The "her" inside "other" is a suffix fragment; the "her" in "where" is an internal sequence; the "her" at the start of "hero" is a prefix. These positions carry different information, and boundary markers let FastText model them separately.

The practical implication is significant. Consider morphologically related pairs like "run" and "runner." With boundary markers, "<run>" and "<runner>" share 3-grams "<ru," "run," "un>" (for the shorter word) and "<ru," "run," "unn," "nn," "nne," "ner," "er>" (for the longer word). Several markers align: the prefix fragments reflect that both words start the same way, and the "run" interior fragment captures the shared root. This shared structure is exactly what we want the model to exploit.

The number of n-grams a word generates grows with its length. Short words have fewer substrings, which gives them less representational capacity. Long words have more, which helps with morphologically complex forms that pack meaning into their length.

The roughly linear scaling is important. It means that the number of parameters used to represent a word grows proportionally with the word's length, which aligns well with the intuition that longer words carry more information per token. Short words like "run" (3 characters) get 9 n-grams; longer words like "appreciation" (12 characters) get around 45. This proportionality is not accidental: the architects of FastText chose the extraction range specifically to maintain this relationship between word complexity and representational capacity.

The distribution by n-gram size also matters. Shorter n-grams (3 and 4 characters) appear in many words, so their embeddings are updated frequently during training and tend to capture broad morphological patterns. Longer n-grams (5 and 6 characters) are more specific, appearing in fewer words, so they capture finer-grained structure. Together, the range from 3 to 6 provides both broad pattern recognition and fine-grained discrimination.

With the decomposition defined, we need a mechanism to combine n-gram embeddings into a single word vector. FastText uses the simplest possible aggregation: summation. Every n-gram has its own learnable embedding vector in a shared table, and a word's representation is the sum of those vectors.

For any word $w$, FastText defines:

where:

• $\mathbf{v}_w$: the final embedding vector for word $w$, used during training and inference
• $\mathbf{z}_w$: an optional word-level embedding for words that appear in the vocabulary (omitted for out-of-vocabulary words)
• $G_w$: the set of all character n-grams extracted from word $w$ using the boundary-marked extraction procedure
• $\mathbf{z}_g$: the learned embedding vector for n-gram $g$, shared across every word that contains that substring

Why sum rather than average, concatenate, or use a learned aggregation? The sum is not arbitrary. The skip-gram training objective involves computing a dot product between the center word representation and context word representations. With summation, the dot product distributes over the sum:

This means that the score for a (center, context) pair decomposes into a sum of scores from individual n-grams. Each n-gram embedding receives a gradient that is directly proportional to how well that n-gram aligns with the context, which makes the gradient computation straightforward and the learning signal interpretable.

The critical design decision is that $\mathbf{z}_g$ is shared. The n-gram "run" appears in "run," "running," "runner," "outrun," and dozens of other words. Every time any of these words appears in training, the gradient flows back to update the embedding for "run." This sharing makes subword embeddings an efficient form of information pooling: common morphological patterns accumulate training signal from every word that contains them, making the embedding for each n-gram far better estimated than it would be if each occurrence were treated independently.

For an out-of-vocabulary word, $\mathbf{z}_w$ simply does not exist. The entire representation comes from $\sum_{g \in G_w} \mathbf{z}_g$. This is what enables zero-shot generalization to unseen words: as long as the word can be decomposed into familiar n-grams, a meaningful embedding can be constructed. No special case handling is needed; the architecture handles OOV words through the same summation mechanism it uses for known words.

The following class implements this representation, including the OOV case:

Even before training, morphologically related words exhibit positive similarity because their n-gram sets overlap. "Runing" (a typo) retains most of "running"'s n-grams, so even with random weights the two are already more similar than "run" and "swim." Training amplifies this structural signal into meaningful semantic proximity. The pre-training similarity is a measure of structural relatedness; post-training similarity reflects semantic relatedness as well.

This is a useful diagnostic. If you initialize a FastText model with random weights and compute cosine similarities, morphological relatives should already show positive correlation. If they do not, something is wrong with the n-gram extraction. The structural bias is a feature, not a bug: it gives gradient descent a better starting point than purely random initialization, which is part of why FastText converges faster than word-level models on morphologically complex vocabularies.

FastText adopts Word2Vec's skip-gram framework: predict context words from center words. The modification is entirely in how the center word is represented. Instead of a single embedding lookup, the center word's representation is computed as a sum over its n-gram embeddings. This substitution is the only change to the original skip-gram architecture.

The compatibility score between center word $w_t$ and context word $w_c$ is:

where:

• $s(w_t, w_c)$: the real-valued compatibility score (higher values encourage co-occurrence)
• $G_{w_t}$: the set of n-grams for the center word $w_t$
• $\mathbf{z}_g$: the n-gram embedding for subword $g$
• $\mathbf{v}_{w_c}$: the context embedding for the context word $w_c$ (a separate embedding table from the n-gram table)

This formula is equivalent to first summing all n-gram embeddings of $w_t$ and then computing a single dot product with $\mathbf{v}_{w_c}$. Writing it as a sum of dot products makes the gradient computation explicit: every n-gram in $G_{w_t}$ receives a gradient proportional to how well the n-gram aligns with the context embedding. The gradient for $\mathbf{z}_g$ is the same regardless of which specific word $w_t$ caused the update, which is what lets knowledge flow across all words sharing that n-gram.

Computing the full softmax over all vocabulary words is prohibitively slow when vocabularies contain hundreds of thousands of words. FastText uses negative sampling, the same approximation as Word2Vec. For each positive (center, context) pair, the model samples $k$ random negative words from the vocabulary and trains a binary classifier to distinguish the true pair from the fake ones.

The core idea behind negative sampling is this: instead of asking "what is the probability of this context word given the center word, compared to all other words?", we ask "can the model tell the difference between an observed co-occurrence pair and a randomly drawn pair?" This binary classification task is much cheaper and, empirically, produces embeddings of comparable quality.

The loss for a single training example is:

where:

• $\mathcal{L}$: the per-example objective, which we maximize (equivalently, we minimize $-\mathcal{L}$)
• $\sigma(x) = \frac{1}{1+e^{-x}}$: the sigmoid function mapping scores to probabilities in $[0, 1]$
• $\log \sigma(s(w_t, w_c))$: the positive term, rewarding high scores for observed pairs
• $k$: the number of negative samples per positive example (typically 5 to 10)
• $w_{n_i}$: the $i$-th randomly sampled negative context word
• $\log \sigma(-s(w_t, w_{n_i}))$: the negative term, penalizing high scores for random pairs

Maximizing $\mathcal{L}$ pushes the model to assign high scores to real co-occurrence pairs and low scores to random ones. Because the center word representation is a sum over n-gram embeddings, each gradient update trains one word embedding and all n-gram embeddings that appear in the center word simultaneously. A single training step on "running" updates the embeddings for "<ru," "run," "unn," "nni," "nin," "ing," "ng>," and every other n-gram in the word. This parameter sharing is the mechanism that makes FastText sample-efficient: every observation of any word containing a morpheme contributes to refining the morpheme's embedding.

The negative sampling distribution matters. FastText follows Word2Vec in using a smoothed unigram distribution raised to the power $\frac{3}{4}$:

where $f(w)$ is the word's frequency in the training corpus. Raising to $\frac{3}{4}$ raises the probability of sampling rare words relative to what a pure frequency-proportional distribution would give. This prevents the most common words from dominating the negative samples, which would make training ignore rare words entirely.

The following implementation brings the architecture to life. We maintain a matrix of n-gram embeddings and a separate matrix of context (output) embeddings for vocabulary words. Forward pass: sum n-gram embeddings for the center word, compute dot products with context embeddings. Backward pass: distribute gradients back to each contributing n-gram.

The train_pair method is the heart of the implementation. Notice that the gradient for the center word, grad_center, is computed once and then applied identically to every n-gram embedding in the word. This is the mathematical consequence of the sum-of-embeddings representation: the gradient of the loss with respect to each $\mathbf{z}_g$ is the same vector, because each n-gram contributes equally to the dot product with the context embedding. In practice, FastText's actual C++ implementation accumulates this gradient and applies it in a single matrix operation for efficiency, but the mathematical meaning is identical.

We train on a small corpus specifically designed to surface morphological relationships. Each sentence places a root form near its inflected variants, so the model must learn that "run," "running," and "runner" are contextually interchangeable:

After training, we can examine the similarity structure to verify that the model learned morphological groupings. The key test is whether words from the same morphological family (all forms of "run," all forms of "swim," and so on) ended up near each other, while words from different families remained distant.

The post-training matrix reveals exactly the structure the architecture was designed to produce. Within each morphological family, similarities are high. Across families, they drop. The white grid lines in the heatmap visually separate the four groups (run family, swim family, walk family, and the pair of adverbs), making the block structure apparent. The n-gram sharing is the mechanism, but training is what turns shared structure into meaningful proximity.

The before-training matrix already shows mild positive similarities within families because shared n-grams produce non-zero dot products even with random weights. Training amplifies this pre-existing signal rather than creating it from scratch, which is one reason FastText converges faster than models that start from purely random per-word embeddings.

The OOV case is where FastText's design pays off most dramatically. Let's test words the model never saw during training. For an OOV word, the model computes its embedding by extracting n-grams, looking up any that appear in the n-gram table (which was built from training vocabulary words), and summing their embeddings. N-grams not in the table are simply skipped.

A PCA projection of the trained embeddings shows whether morphological families ended up near each other in the full embedding space. PCA finds the directions of maximum variance in the embedding matrix and projects each vector onto the first two principal components, producing a 2D view that preserves as much structure as possible from the original 30-dimensional space.

OOV words land within or near the correct family regions despite never having been seen during training. The PCA projection confirms that the n-gram sharing mechanism is doing meaningful work: structural similarity in character sequences translates into proximity in the learned embedding space. When the model has never seen "runing" as a complete token, it can still place it near "run," "running," and "runner" because most of its n-grams are familiar from training on related words.

PCA preserves some structure and loses some. The first two principal components capture the directions of maximum variance, which in this case align with the between-family separation axis and within-family variation. The actual embedding space is 30-dimensional, and some structure that exists there is invisible in the 2D projection. The clusters you see are real, but the distances between clusters in 2D are not perfectly representative of distances in 30 dimensions.

Our implementation maintains one embedding vector per unique n-gram. For a small corpus that is fine, but for production-scale vocabularies this fails. A vocabulary of 100,000 words with n-grams of length 3 to 6 can generate several million unique n-grams. At 300 dimensions and 4 bytes per float, that exceeds 2 GB of memory before counting word embeddings. No single inference machine can load this into RAM efficiently.

FastText addresses this with hash bucketing. Rather than allocating a unique embedding for each n-gram, the model hashes each n-gram to one of $B$ buckets and stores an embedding per bucket. The memory usage is then exactly $B \times d \times 4$ bytes regardless of how many unique n-grams exist in the vocabulary.

The hashing trick was popularized in machine learning by the "feature hashing" or "hashing trick" paper, and FastText applies it directly to n-gram storage. The approach works because hash collisions are rare when the number of buckets is large relative to the number of distinct n-grams, and when collisions do occur, they cause mild accuracy degradation rather than catastrophic failures. Two unrelated n-grams sharing a bucket means their embeddings are conflated, which adds noise but does not prevent learning.

Hashing introduces a modest accuracy trade-off: colliding n-grams share embeddings, which can conflate unrelated substrings. In practice, at 2 million buckets, collision rates for typical English are well below 5%, and empirical benchmarks show negligible impact on downstream task quality. The memory savings are dramatic and allow FastText models to be deployed on machines with limited RAM, including mobile devices and embedded systems.

One practical consideration when choosing the bucket count: the buckets are always allocated, even if many are empty. So choosing 2 million buckets commits to $2{,}000{,}000 \times 300 \times 4 \approx 2.4$ GB of RAM regardless of whether the vocabulary is large or small. For small vocabularies, a smaller bucket count is appropriate. The rule of thumb from the paper's experiments is to use at least as many buckets as there are unique n-grams in the training vocabulary, which avoids collisions almost entirely while still bounding memory at a fixed size.

English morphology is relatively shallow: a handful of common suffixes cover most of the grammatical variation, and the number of distinct forms per word is modest. The real demonstration of FastText's power is on languages where morphology is far more productive, where the number of distinct word forms per root can run into the thousands.

German builds compound nouns without spaces. "Lebensversicherungsgesellschaft" (life insurance company) joins three meaningful roots: "Leben" (life), "Versicherung" (insurance), and "Gesellschaft" (company). A Word2Vec model will never see this exact compound in most corpora. FastText, however, will share n-grams between the compound and its components, encoding the compositional meaning in the embedding automatically. The n-gram "leben" (from "Leben") will match both the standalone word and the compound prefix. The n-gram "gesellschaft" will match the component and the compound suffix.

This compositional encoding is not perfect. FastText does not know where one root ends and another begins inside the compound. But it does not need to: the n-gram overlap gives a rough approximation of the compositional structure, which is enough to produce useful embeddings even for compounds that appear only once in a billion-word corpus.

Every German compound shares a substantial fraction of its n-grams with its component roots. "Krankenhaus" (hospital) shares about 60% of its n-grams with "Krank" (sick) and "Haus" (house). A FastText model trained on a German corpus will place "Krankenhaus" in a region of embedding space influenced by both components' embeddings, even if the compound never appears in training. The embedding captures compositional meaning for free, without any explicit morphological analyzer or dictionary lookup.

The same logic applies across a wide range of languages and linguistic phenomena:

• Turkish and Finnish (agglutinative morphology): These languages stack grammatical suffixes onto roots in long chains. "Evlerinizden" (from your houses) in Turkish shares n-grams with "ev" (house) and related suffixed forms. FastText handles new suffix combinations naturally because it has seen the individual suffix n-grams many times during training.

• Arabic and Hebrew (templatic morphology): These languages use a three-consonant root system where a single root interleaves with vowel patterns to produce different word classes and tenses. Character n-grams capture the consonant sequences that encode root meaning, even though the vowel patterns change between forms.

• Any language with productive affixation: Medical and scientific terminology across all languages uses Latin and Greek roots that recombine in predictable ways. "Electroencephalography" shares n-grams with "electro," "encephalo," "graph," and their Latin derivatives, giving FastText a way to anchor even unfamiliar coinages in a meaningful region of embedding space.

To make the cross-lingual benefit concrete, consider what happens when a FastText model trained on Turkish encounters the word "evlerinizden" (from your houses), which might not appear in the training corpus at all. The word decomposes into the root "ev" (house), followed by the plural suffix "-ler," the possessive suffix "-iniz" (your), and the ablative case suffix "-den" (from). Each suffix appears in hundreds of other Turkish words in training. The n-grams "<evl," "evle," "vler," "leri," "erin," "riniz" and others appear in related words. The embedding FastText assigns to this never-seen word will be a blend of embeddings it has seen from "ev," from other words ending in "-den," from other words containing "iniz," and so on. The result is imperfect but meaningful, far better than the zero-vector fallback that Word2Vec would provide.

This is also why FastText was a significant advance for low-resource language processing. Languages with small Wikipedia corpora but rich morphology, like Swahili or Icelandic, have relatively few unique surface forms in any given training corpus, but those forms cover a much larger space of meanings through morphological combination. FastText exploits this structure to learn embeddings that generalize across the morphological space, making it a practical choice even when data is scarce.

Training FastText from scratch requires a large corpus and substantial compute time. For most applications, a better starting point is one of the pretrained models that Meta (Facebook AI Research) distributes publicly. These cover 157 languages and were trained on Wikipedia and Common Crawl text, giving broad coverage across both formal and informal registers.

The fasttext Python package provides a clean interface for loading and querying these models. The steps below show the main operations you will use in practice:

The pretrained models use the full production settings: 300-dimensional embeddings, 2 million hash buckets, n-gram range 3 to 6, and training on datasets orders of magnitude larger than our toy example. The nearest-neighbor results from these models reflect semantic and morphological relationships captured from billions of tokens of text.

One practical note: the .bin format includes the full n-gram table needed for OOV inference. If you only need embeddings for known vocabulary words, you can use the .vec format, which is a simple text file of word-to-vector mappings. The .vec format is much smaller (a few hundred MB versus several GB) but loses OOV capability entirely.

FastText advances Word2Vec, but it inherits one critical limitation from the static embedding paradigm and introduces one new concern of its own.

Every word has exactly one embedding regardless of which sentence it appears in. The word "bank" receives the same vector whether you write "the river bank" or "the bank account." FastText has no mechanism to distinguish these senses because the embedding is computed from character n-grams alone, with no awareness of the surrounding text. This is not a flaw in FastText specifically but a property of all static embedding methods: the representation is computed once per word, offline, before any inference happens.

The problem is real and consequential. Polysemous words are extremely common in English and other natural languages. WordNet catalogs an average of 2.6 senses per noun. For words in the long tail of sense distributions, a single averaged vector is a poor representation of any individual sense. Downstream models that use static embeddings must implicitly recover the correct sense from context, which places an extra burden on the task-specific model and limits how much of the semantic signal the embeddings can provide.

The long-term solution to context independence came from a different direction. ELMo, introduced in 2018, used bidirectional LSTMs to produce word representations that depend on the entire sentence context. The same word in different sentences produces different vectors. BERT extended this with the transformer architecture, producing context-sensitive representations that captured word senses, syntactic roles, coreference, and other discourse-level phenomena. FastText embeddings and BERT contextual embeddings are designed for different situations: FastText is fast, memory-efficient, and well-suited to morphologically rich languages, while BERT produces richer representations at the cost of much higher compute and memory requirements.

The n-gram mechanism cannot distinguish meaningful morphological segments from arbitrary character sequences. "xyzqwk" produces a non-zero embedding by summing its n-gram vectors, even though no human would recognize it as a word from any known language. Applications that rely on zero-norm vectors to signal unknown tokens must add explicit out-of-vocabulary detection logic. In practice this is an inconvenience rather than a fundamental problem, but it matters when deploying FastText in a pipeline that needs to flag anomalous inputs.

Related to this: FastText embeddings for very short words (two or three characters) can be low quality because short words have few n-grams. The word "a" with the default minimum of 3 and maximum of 6 produces just the single n-gram "<a>," which gives it almost no representational capacity. Short function words in general tend to be handled less well by FastText than by word-level models, which is the reverse of the situation for long morphologically complex words.

Even with hashing, storing 2 million 300-dimensional float vectors requires about 2.4 GB for the n-gram table alone. The full binary models released by Meta can exceed 7 GB when word-level embeddings are included. This makes cold-start deployment on mobile devices or in memory-constrained environments challenging. Techniques like product quantization (compressing each embedding vector into a small code) and model distillation (training a smaller model to mimic the larger one) are commonly used to reduce the footprint.

The 300-dimensional default also means that every embedding operation involves multiplying or adding 300-dimensional vectors, which is fast on modern hardware but adds up when processing millions of documents in a real-time pipeline. For latency-sensitive applications, smaller dimensionality (50 or 100 dimensions) often provides a good trade-off between speed and accuracy.

Character n-grams assume that subword character sequences carry meaning. This assumption is sound for alphabetic and syllabic writing systems, where characters map to sounds and sounds map to meaning. But for Chinese, Japanese kanji, and other logographic systems, individual characters often map directly to morphemes or entire words. The n-gram of two adjacent characters may be entirely meaningless as a linguistic unit, or it may be a compound word, with no predictable relationship between character position and meaning. Chinese NLP typically uses word-level tokenization (with dedicated Chinese word segmentation tools) or character-level models rather than FastText's fixed-length n-gram approach. Specialized methods exist for these scripts, and FastText's published results for Chinese and Japanese are generally weaker than for alphabetic languages.

Despite these limitations, FastText occupies an important and durable niche. For downstream tasks on morphologically rich languages, for applications where OOV robustness matters, and for deployment scenarios where a compact static embedding model is preferred over a large contextual model, FastText consistently outperforms Word2Vec with modest additional computational cost. Pretrained FastText models for 157 languages are freely available from Meta, making it easy to use without training from scratch.

FastText extends Word2Vec by replacing word-level atomic representations with sums of character n-gram embeddings. This single architectural change yields three practical benefits: handling of out-of-vocabulary words, morphological awareness through shared n-gram structure, and less brittleness to typos and noisy text. The training objective remains skip-gram with negative sampling, but gradients now flow back to all n-gram embeddings in a center word, allowing shared learning across morphological variants. The hashing trick bounds memory usage at a fixed bucket count regardless of vocabulary size. This makes deployment practical at scale.

Key takeaways from this chapter:

• Subword representation: Each word is decomposed into character n-grams (default lengths 3 to 6), with boundary markers separating positional from interior contexts
• Embedding computation: A word's embedding is the sum of its n-gram embeddings, plus an optional word-level term for known vocabulary words
• OOV handling: Any word can receive an embedding by summing whatever n-gram embeddings were learned, enabling zero-shot generalization to unseen words, typos, and morphological variants
• Morphological awareness: Words sharing the same root share n-grams, so morphological variants naturally cluster in embedding space even for words never seen during training
• Hash bucketing: N-grams are hashed to a fixed number of buckets to control memory usage, trading minor accuracy for large memory savings at production scale
• Language applicability: FastText excels on morphologically rich alphabetic languages but has limitations for logographic scripts like Chinese
• Context independence: Like all static embeddings, FastText cannot distinguish word senses from context, which limits its ability to represent polysemous words accurately

The next chapter on embedding evaluation will give us tools to measure these properties quantitatively: how to test similarity, how to assess OOV coverage, and how to detect when embedding quality is insufficient for a downstream task. The chapter after that introduces contextual embeddings with ELMo, which addresses the context-independence limitation at the cost of higher computational complexity.

The key parameters for FastText training are:

• min_n: Minimum n-gram length. The default value of 3 captures short prefixes and suffixes. Setting it to 2 includes bigrams like "th" and "ed," which helps for very short words but roughly doubles the n-gram count and can add noisy, low-information substrings to many embeddings.

• max_n: Maximum n-gram length. The default value of 6 balances morpheme capture with vocabulary size. For agglutinative languages with longer suffixes (Turkish, Finnish, Hungarian), values of 7 or 8 improve coverage of long morphological patterns.

• bucket: Number of hash buckets for n-gram storage. The default of 2,000,000 keeps collision rates below 5% for typical English vocabularies. Reducing this saves memory but increases collisions; values below 100,000 noticeably degrade embedding quality. For small datasets, 500,000 to 1,000,000 is often sufficient.

• dim: Embedding dimensionality. Values of 100 to 300 are standard. Lower dimensions train faster and generalize better on small datasets; higher dimensions capture more nuance but require more data to fill the representational space without overfitting.

• epoch: Training passes over the corpus. The range of 5 to 25 covers most use cases. More epochs help on smaller datasets; for very large corpora (Common Crawl scale), even 1 epoch provides enough training signal for high-quality embeddings.

• lr: Learning rate. The default of 0.05 works well with the linear decay schedule FastText uses by default, which reduces the learning rate linearly to zero over training. Higher values speed training but risk overshooting on small or noisy corpora.

• neg: Number of negative samples per positive pair. Values of 5 to 10 are standard. Higher values improve training quality for rare words at the cost of slower training. For very large corpora, 5 negative samples is usually sufficient.

• ws: Context window size. The default of 5 captures sentence-level co-occurrence. Smaller windows capture tight syntactic relationships (useful for part-of-speech tagging and named entity recognition); larger windows capture broader topical associations (useful for document classification and information retrieval).

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about FastText and character n-gram embeddings.