CBOW Model › Worked Example: Full Training Step
Language AI HandbookAll chapters CBOW Model › Worked Example: Full Training StepOne section · Sections in this chapter9 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us trace through a complete CBOW training step for the example sentence, confirming that all the pieces fit together.
Consider the sentence "The quick brown fox jumps over the lazy dog" with target word "fox" and context ["quick", "brown", "jumps", "over"] (window size 2). We will perform one complete gradient descent step and verify the loss decreases. This walkthrough connects the mathematical derivations above to the concrete numerical operations that happen in every training iteration.
The procedure has four stages. First, the forward pass computes from the context embeddings and then computes the probability distribution over the vocabulary using the output matrix and softmax. Second, the loss measures how far the current distribution is from the ideal (all probability on "fox"). Third, the backward pass computes gradients for all parameters. Fourth, the update subtracts the scaled gradients from the current parameters.
# Trace a single training step end-to-end
np.random.seed(7)
V_small = len(small_vocab)
d_small = 3 # Tiny embedding for visibility
W_step = np.random.randn(V_small, d_small) * 0.3
W_prime_step = np.random.randn(d_small, V_small) * 0.3
lr = 0.1
target = "fox"
context_ex = ["quick", "brown", "jumps", "over"]
# --- Before update ---
h_before, probs_before = cbow_forward(
context_ex, W_step, W_prime_step, word_to_idx
)
loss_before = compute_loss(target, probs_before, word_to_idx)
# Backward pass
dW_step, dWp_step = cbow_backward(
context_ex,
target,
h_before,
probs_before,
W_step,
W_prime_step,
word_to_idx,
)
# Update
W_step -= lr * dW_step
W_prime_step -= lr * dWp_step
# --- After update ---
h_after, probs_after = cbow_forward(
context_ex, W_step, W_prime_step, word_to_idx
)
loss_after = compute_loss(target, probs_after, word_to_idx)Full CBOW Training Step ================================================== Context: ['quick', 'brown', 'jumps', 'over'] Target: 'fox' Word P before P after Change --------------------------------------------- the 0.1355 0.1352 -0.0003 quick 0.1209 0.1214 +0.0005 brown 0.1383 0.1382 -0.0000 fox 0.1283 0.1299 +0.0016 <-- target jumps 0.1242 0.1238 -0.0004 over 0.1102 0.1104 +0.0002 lazy 0.1139 0.1138 -0.0001 dog 0.1287 0.1273 -0.0013 Loss before: 2.0535 Loss after: 2.0409 Loss decrease: 0.0126
After a single gradient step, the probability assigned to "fox" increases while probabilities for incorrect words decrease. The loss decreases, confirming that the gradient step moved the parameters in the right direction. This is the fundamental loop of CBOW training: observe the prediction error, compute gradients that would reduce the error, and take a small step in that direction. Repeat this across millions of training examples, and the embeddings gradually organize themselves so that words with similar distributions end up in similar regions of the space.
Notice that the improvement from a single step is modest: the probability of "fox" might increase from 0.10 to 0.12, while the loss decreases by a fraction of a nit. This is expected and desirable. Large updates would overfit to individual training examples. The embedding space is shaped by the aggregate of millions of small updates, each nudging the representations slightly toward the global structure encoded in the corpus.
Visualizing Trained Embeddings
After training, the embedding matrix contains meaningful word representations. Even on a small corpus, words that appear in similar contexts cluster together in embedding space. We can project the 8-dimensional embeddings down to 2D using PCA to visualize these relationships.
Principal Component Analysis (PCA) finds the directions of maximum variance in the embedding space and projects the data onto them. The first principal component captures the most variance, and the second captures the next most. A 2D PCA projection preserves as much structure as possible while making the data visualizable. The resulting scatter plot does not show the full geometry (much information is lost in reducing 8 dimensions to 2), but it reveals the broad clustering structure that the model has learned.
Even with this small corpus and brief training, words with similar grammatical roles and contextual distributions tend to cluster together. The nouns "fox" and "dog" should occupy nearby positions because both appear as subjects and objects in similar sentence structures. Adjectives like "quick," "brown," and "lazy" should cluster because they all appear before nouns. With a larger corpus and more training, these clusters become much tighter and semantically meaningful. The embeddings capture the statistical regularities of the text, placing words in positions that reflect their distributional behavior.