FastText › The FastText Training Objective

FastText adopts Word2Vec's skip-gram framework: predict context words from center words. The modification is entirely in how the center word is represented. Instead of a single embedding lookup, the center word's representation is computed as a sum over its n-gram embeddings. This substitution is the only change to the original skip-gram architecture.

The compatibility score between center word $w_t$ and context word $w_c$ is:

$$s(w_t, w_c) = \sum_{g \in G_{w_t}} \mathbf{z}_g^T \mathbf{v}_{w_c}$$

where:

• $s(w_t, w_c)$: the real-valued compatibility score (higher values encourage co-occurrence)
• $G_{w_t}$: the set of n-grams for the center word $w_t$
• $\mathbf{z}_g$: the n-gram embedding for subword $g$
• $\mathbf{v}_{w_c}$: the context embedding for the context word $w_c$ (a separate embedding table from the n-gram table)

This formula is equivalent to first summing all n-gram embeddings of $w_t$ and then computing a single dot product with $\mathbf{v}_{w_c}$. Writing it as a sum of dot products makes the gradient computation explicit: every n-gram in $G_{w_t}$ receives a gradient proportional to how well the n-gram aligns with the context embedding. The gradient for $\mathbf{z}_g$ is the same regardless of which specific word $w_t$ caused the update, which is what lets knowledge flow across all words sharing that n-gram.

Computing the full softmax over all vocabulary words is prohibitively slow when vocabularies contain hundreds of thousands of words. FastText uses negative sampling, the same approximation as Word2Vec. For each positive (center, context) pair, the model samples $k$ random negative words from the vocabulary and trains a binary classifier to distinguish the true pair from the fake ones.

The core idea behind negative sampling is this: instead of asking "what is the probability of this context word given the center word, compared to all other words?", we ask "can the model tell the difference between an observed co-occurrence pair and a randomly drawn pair?" This binary classification task is much cheaper and, empirically, produces embeddings of comparable quality.

The loss for a single training example is:

$$\mathcal{L} = \log \sigma\bigl(s(w_t, w_c)\bigr) + \sum_{i=1}^{k} \log \sigma\bigl(-s(w_t, w_{n_i})\bigr)$$

where:

• $\mathcal{L}$: the per-example objective, which we maximize (equivalently, we minimize $-\mathcal{L}$)
• $\sigma(x) = \frac{1}{1+e^{-x}}$: the sigmoid function mapping scores to probabilities in $[0, 1]$
• $\log \sigma(s(w_t, w_c))$: the positive term, rewarding high scores for observed pairs
• $k$: the number of negative samples per positive example (typically 5 to 10)
• $w_{n_i}$: the $i$-th randomly sampled negative context word
• $\log \sigma(-s(w_t, w_{n_i}))$: the negative term, penalizing high scores for random pairs

Maximizing $\mathcal{L}$ pushes the model to assign high scores to real co-occurrence pairs and low scores to random ones. Because the center word representation is a sum over n-gram embeddings, each gradient update trains one word embedding and all n-gram embeddings that appear in the center word simultaneously. A single training step on "running" updates the embeddings for "<ru," "run," "unn," "nni," "nin," "ing," "ng>," and every other n-gram in the word. This parameter sharing is the mechanism that makes FastText sample-efficient: every observation of any word containing a morpheme contributes to refining the morpheme's embedding.

The negative sampling distribution matters. FastText follows Word2Vec in using a smoothed unigram distribution raised to the power $\frac{3}{4}$:

$$P(w) \propto f(w)^{3/4}$$

where $f(w)$ is the word's frequency in the training corpus. Raising to $\frac{3}{4}$ raises the probability of sampling rare words relative to what a pure frequency-proportional distribution would give. This prevents the most common words from dominating the negative samples, which would make training ignore rare words entirely.