Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

FastText › From N-grams to Word Vectors

Michael BrenndoerferUpdated March 10, 20265 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 26FastText › From N-grams to Word Vectors

Reading section 4 of 15 · From N-grams to Word Vectors · 5 minSection 4 of 15 · 5 min

Section 4 of 15 · 5 min read

FastText › From N-grams to Word VectorsOne section · Sections in this chapter4 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

With the decomposition defined, we need a mechanism to combine n-gram embeddings into a single word vector. FastText uses the simplest possible aggregation: summation. Every n-gram has its own learnable embedding vector in a shared table, and a word's representation is the sum of those vectors.

For any word $w$ , FastText defines:

\mathbf{v}_w = \mathbf{z}_w + \sum_{g \in G_w} \mathbf{z}_g

where:

$\mathbf{v}_w$ : the final embedding vector for word $w$ , used during training and inference
$\mathbf{z}_w$ : an optional word-level embedding for words that appear in the vocabulary (omitted for out-of-vocabulary words)
$G_w$ : the set of all character n-grams extracted from word $w$ using the boundary-marked extraction procedure
$\mathbf{z}_g$ : the learned embedding vector for n-gram $g$ , shared across every word that contains that substring

Why sum rather than average, concatenate, or use a learned aggregation? The sum is not arbitrary. The skip-gram training objective involves computing a dot product between the center word representation and context word representations. With summation, the dot product distributes over the sum:

\mathbf{v}_w^T \mathbf{v}_c = \left(\sum_{g \in G_w} \mathbf{z}_g\right)^T \mathbf{v}_c = \sum_{g \in G_w} \mathbf{z}_g^T \mathbf{v}_c

This means that the score for a (center, context) pair decomposes into a sum of scores from individual n-grams. Each n-gram embedding receives a gradient that is directly proportional to how well that n-gram aligns with the context, which makes the gradient computation straightforward and the learning signal interpretable.

The critical design decision is that $\mathbf{z}_g$ is shared. The n-gram "run" appears in "run," "running," "runner," "outrun," and dozens of other words. Every time any of these words appears in training, the gradient flows back to update the embedding for "run." This sharing makes subword embeddings an efficient form of information pooling: common morphological patterns accumulate training signal from every word that contains them, making the embedding for each n-gram far better estimated than it would be if each occurrence were treated independently.

For an out-of-vocabulary word, $\mathbf{z}_w$ simply does not exist. The entire representation comes from $\sum_{g \in G_w} \mathbf{z}_g$ . This is what enables zero-shot generalization to unseen words: as long as the word can be decomposed into familiar n-grams, a meaningful embedding can be constructed. No special case handling is needed; the architecture handles OOV words through the same summation mechanism it uses for known words.

The following class implements this representation, including the OOV case:

In[11]:

Code

class FastTextEmbeddings:
    """
    FastText-style word embeddings using character n-grams.
    This is a conceptual implementation for understanding the architecture.
    """

    def __init__(self, embedding_dim=50, min_n=3, max_n=6):
        self.embedding_dim = embedding_dim
        self.min_n = min_n
        self.max_n = max_n
        self.word_embeddings = {}  # z_w: word-level embeddings for known vocab
        self.ngram_embeddings = {}  # z_g: n-gram embeddings shared across words

    def _get_ngrams(self, word):
        return get_ngrams(word, self.min_n, self.max_n)

    def _init_ngram(self, ng):
        if ng not in self.ngram_embeddings:
            self.ngram_embeddings[ng] = (
                np.random.randn(self.embedding_dim) * 0.01
            )

    def get_embedding(self, word):
        """
        Compute embedding as sum of n-gram embeddings plus
        optional word-level embedding for known vocab words.
        """
        ngs = self._get_ngrams(word)
        for ng in ngs:
            self._init_ngram(ng)

        # Start with word-level embedding if available
        if word in self.word_embeddings:
            vec = self.word_embeddings[word].copy()
        else:
            vec = np.zeros(self.embedding_dim)

        # Add all n-gram embeddings
        for ng in ngs:
            vec += self.ngram_embeddings[ng]

        return vec


np.random.seed(42)
ft = FastTextEmbeddings(embedding_dim=50)

# Seed some known words into the word-level embedding table
for w in ["run", "running", "runner", "swim", "swimming"]:
    ft.word_embeddings[w] = np.random.randn(50) * 0.1

# Compute embeddings for known and out-of-vocabulary words
emb_running = ft.get_embedding("running")  # in vocab
emb_runs = ft.get_embedding("runs")  # OOV: inflected form
emb_runing = ft.get_embedding("runing")  # OOV: typo
emb_swim = ft.get_embedding("swim")  # in vocab

Out[12]:

Console

Embedding similarities (random initialization, no training):
-------------------------------------------------------
  'run'  vs 'runs'    (inflection): 0.1958
  'run'  vs 'running' (inflection): 0.0469
  'running' vs 'runing' (typo):     0.0091
  'run'  vs 'swim'    (unrelated):  0.0584

Even with random weights, shared n-grams produce non-trivial
similarity between morphologically related words.

Even before training, morphologically related words exhibit positive similarity because their n-gram sets overlap. "Runing" (a typo) retains most of "running"'s n-grams, so even with random weights the two are already more similar than "run" and "swim." Training amplifies this structural signal into meaningful semantic proximity. The pre-training similarity is a measure of structural relatedness; post-training similarity reflects semantic relatedness as well.

This is a useful diagnostic. If you initialize a FastText model with random weights and compute cosine similarities, morphological relatives should already show positive correlation. If they do not, something is wrong with the n-gram extraction. The structural bias is a feature, not a bug: it gives gradient descent a better starting point than purely random initialization, which is part of why FastText converges faster than word-level models on morphologically complex vocabularies.