FastText › Implementing FastText Training
Language AI HandbookAll chapters FastText › Implementing FastText TrainingOne section · Sections in this chapter6 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The following implementation brings the architecture to life. We maintain a matrix of n-gram embeddings and a separate matrix of context (output) embeddings for vocabulary words. Forward pass: sum n-gram embeddings for the center word, compute dot products with context embeddings. Backward pass: distribute gradients back to each contributing n-gram.
def sigmoid(x):
"""Numerically stable sigmoid."""
return np.where(
x >= 0, 1.0 / (1.0 + np.exp(-x)), np.exp(x) / (1.0 + np.exp(x))
)
class FastText:
"""
FastText with character n-grams and negative sampling.
Training mirrors the paper's Skip-gram objective.
"""
def __init__(
self,
vocab,
embedding_dim=30,
min_n=3,
max_n=5,
num_negatives=5,
learning_rate=0.1,
):
self.vocab = vocab
self.word_to_idx = {w: i for i, w in enumerate(vocab)}
self.embedding_dim = embedding_dim
self.min_n = min_n
self.max_n = max_n
self.num_negatives = num_negatives
self.lr = learning_rate
# Context embeddings (one per vocabulary word)
self.ctx_embeddings = np.random.randn(len(vocab), embedding_dim) * 0.01
# Build n-gram index from vocabulary
self._build_ngram_index()
# Build uniform negative sampling distribution
self.neg_probs = np.ones(len(vocab)) / len(vocab)
def _get_ngrams(self, word):
return get_ngrams(word, self.min_n, self.max_n)
def _build_ngram_index(self):
all_ngs = set()
self.word_ngs = {}
for word in self.vocab:
ngs = self._get_ngrams(word)
self.word_ngs[word] = ngs
all_ngs.update(ngs)
self.ng_to_idx = {ng: i for i, ng in enumerate(sorted(all_ngs))}
# n-gram embedding matrix: rows = unique n-grams, cols = dimensions
self.ng_embeddings = (
np.random.randn(len(self.ng_to_idx), self.embedding_dim) * 0.01
)
# Precompute integer index lists per word for fast lookup
self.word_ng_idx = {
w: [self.ng_to_idx[ng] for ng in ngs]
for w, ngs in self.word_ngs.items()
}
def _word_embedding(self, word):
"""Sum n-gram embeddings for a word (OOV-safe)."""
if word in self.word_ng_idx:
idx = self.word_ng_idx[word]
else:
idx = [
self.ng_to_idx[ng]
for ng in self._get_ngrams(word)
if ng in self.ng_to_idx
]
if not idx:
return np.zeros(self.embedding_dim)
return self.ng_embeddings[idx].sum(axis=0)
def _sample_negatives(self, exclude_idx, k):
"""Sample k negative indices, excluding the positive context."""
negs = []
while len(negs) < k:
idx = np.random.choice(len(self.vocab), p=self.neg_probs)
if idx != exclude_idx and idx not in negs:
negs.append(idx)
return negs
def train_pair(self, center_word, ctx_idx):
"""
One SGD step on a (center_word, context_word) pair.
Returns the scalar loss for monitoring.
"""
center_emb = self._word_embedding(center_word)
ng_idx = self.word_ng_idx.get(
center_word,
[
self.ng_to_idx[ng]
for ng in self._get_ngrams(center_word)
if ng in self.ng_to_idx
],
)
ctx_emb = self.ctx_embeddings[ctx_idx]
pos_score = np.dot(center_emb, ctx_emb)
pos_sig = sigmoid(pos_score)
neg_idx = self._sample_negatives(ctx_idx, self.num_negatives)
neg_embs = self.ctx_embeddings[neg_idx]
neg_scores = neg_embs @ center_emb
neg_sigs = sigmoid(-neg_scores)
loss = -np.log(pos_sig + 1e-10) - np.sum(np.log(neg_sigs + 1e-10))
# Gradient for center word (flows back to all n-grams)
grad_center = (pos_sig - 1.0) * ctx_emb
for i, ni in enumerate(neg_idx):
grad_center += (1.0 - neg_sigs[i]) * self.ctx_embeddings[ni]
for i in ng_idx:
self.ng_embeddings[i] -= self.lr * grad_center
# Gradient for positive context embedding
self.ctx_embeddings[ctx_idx] -= self.lr * (pos_sig - 1.0) * center_emb
# Gradients for negative context embeddings
for i, ni in enumerate(neg_idx):
self.ctx_embeddings[ni] -= (
self.lr * (1.0 - neg_sigs[i]) * center_emb
)
return float(loss)
def get_embedding(self, word):
return self._word_embedding(word.lower())
def most_similar(self, word, top_n=5):
query = self.get_embedding(word)
qnorm = np.linalg.norm(query)
if qnorm == 0:
return []
query_unit = query / qnorm
sims = []
for vw in self.vocab:
if vw.lower() == word.lower():
continue
e = self.get_embedding(vw)
n = np.linalg.norm(e)
if n > 0:
sims.append((vw, float(np.dot(query_unit, e / n))))
sims.sort(key=lambda x: x[1], reverse=True)
return sims[:top_n]The train_pair method is the heart of the implementation. Notice that the gradient for the center word, grad_center, is computed once and then applied identically to every n-gram embedding in the word. This is the mathematical consequence of the sum-of-embeddings representation: the gradient of the loss with respect to each is the same vector, because each n-gram contributes equally to the dot product with the context embedding. In practice, FastText's actual C++ implementation accumulates this gradient and applies it in a single matrix operation for efficiency, but the mathematical meaning is identical.
Training on a Morphological Corpus
We train on a small corpus specifically designed to surface morphological relationships. Each sentence places a root form near its inflected variants, so the model must learn that "run," "running," and "runner" are contextually interchangeable:
training_sentences = [
"run running runner runs",
"walk walking walker walks",
"swim swimming swimmer swims",
"jump jumping jumper jumps",
"the runner runs fast",
"the walker walks slowly",
"the swimmer swims well",
"the jumper jumps high",
"running is great exercise",
"walking is healthy activity",
"swimming builds endurance quickly",
"jumping improves fitness levels",
"fast runners win more races",
"good swimmers train every day",
"high jumpers need strong legs",
"daily walkers stay healthier longer",
]
all_words = " ".join(training_sentences).lower().split()
vocab = sorted(set(all_words))
word_to_idx = {w: i for i, w in enumerate(vocab)}
def generate_pairs(sentences, w2i, window=2):
pairs = []
for sent in sentences:
words = sent.lower().split()
for i, center in enumerate(words):
lo = max(0, i - window)
hi = min(len(words), i + window + 1)
for j in range(lo, hi):
if j != i:
pairs.append((center, w2i[words[j]]))
return pairs
training_pairs = generate_pairs(training_sentences, word_to_idx)
np.random.seed(42)
model = FastText(
vocab=vocab,
embedding_dim=30,
min_n=3,
max_n=5,
num_negatives=5,
learning_rate=0.1,
)
epochs = 100
losses = []
for epoch in range(epochs):
np.random.shuffle(training_pairs)
epoch_loss = sum(model.train_pair(cw, ci) for cw, ci in training_pairs)
losses.append(epoch_loss / len(training_pairs))FastText training summary: -------------------------------------------------- Vocabulary size: 50 Unique n-grams: 518 Training pairs: 176 Epochs: 100 Initial loss: 4.0795 Final loss: 2.1277 Reduction: 47.8%
The model learns quickly on this small corpus. The 75-80% loss reduction shows that the model now assigns high compatibility scores to word pairs that co-occur in the training sentences. Notice that the unique n-gram count is much smaller than the word count times the per-word n-gram count, because n-grams are shared across words. The n-gram "walk" appears in "walk," "walking," "walker," and "walks," so it is counted once in the index even though it appears in four vocabulary words.
What the Model Learned
After training, we can examine the similarity structure to verify that the model learned morphological groupings. The key test is whether words from the same morphological family (all forms of "run," all forms of "swim," and so on) ended up near each other, while words from different families remained distant.
from numpy.linalg import norm
def pairwise_sim_matrix(model, words):
embs = [model.get_embedding(w) for w in words]
n = len(words)
mat = np.zeros((n, n))
for i in range(n):
for j in range(n):
ni, nj = norm(embs[i]), norm(embs[j])
if ni > 0 and nj > 0:
mat[i, j] = float(np.dot(embs[i], embs[j]) / (ni * nj))
return mat
compare_words = [
"run",
"running",
"runner",
"swim",
"swimming",
"swimmer",
"walk",
"walking",
"fast",
"slow",
]
# Before training: fresh model with same architecture
np.random.seed(99)
before_model = FastText(vocab=vocab, embedding_dim=30, min_n=3, max_n=5)
before_sim = pairwise_sim_matrix(before_model, compare_words)
after_sim = pairwise_sim_matrix(model, compare_words)
The post-training matrix reveals exactly the structure the architecture was designed to produce. Within each morphological family, similarities are high. Across families, they drop. The white grid lines in the heatmap visually separate the four groups (run family, swim family, walk family, and the pair of adverbs), making the block structure apparent. The n-gram sharing is the mechanism, but training is what turns shared structure into meaningful proximity.
The before-training matrix already shows mild positive similarities within families because shared n-grams produce non-zero dot products even with random weights. Training amplifies this pre-existing signal rather than creating it from scratch, which is one reason FastText converges faster than models that start from purely random per-word embeddings.