CBOW Model: Learning Word Embeddings by Predicting Center Words

CBOW ModelLink Copied

In the previous chapter, we explored Skip-gram, which learns word embeddings by predicting context words from a center word. The Continuous Bag of Words (CBOW) model takes the opposite approach: given the surrounding context words, predict the center word. This architectural inversion leads to different learning dynamics and computational trade-offs.

CBOW was introduced alongside Skip-gram in the original Word2Vec paper by Mikolov et al. (2013). While Skip-gram treats each context position independently, CBOW averages context word embeddings together to make a single prediction per training example. This design choice has practical implications: CBOW trains faster than Skip-gram and performs better on frequent words, while Skip-gram excels at representing rare words.

This chapter covers the CBOW architecture from the ground up. We'll work through the mathematics, implement the model from scratch, and understand when to choose CBOW over Skip-gram.

The Core Idea: Predicting Words from ContextLink Copied

Imagine reading a sentence with a word missing: "The quick brown ___ jumps over the lazy dog." Given the surrounding context ("quick," "brown," "jumps," "over"), you can likely guess the missing word is "fox." This fill-in-the-blank task is exactly what CBOW learns to do.

The name "Continuous Bag of Words" comes from two properties:

1. Continuous: The model uses continuous-valued vectors (embeddings) rather than discrete word representations
2. Bag of Words: The context words are treated as an unordered set, averaging their embeddings together regardless of position

This single training example captures the essence of CBOW: given the surrounding words, predict what goes in the middle. Compare this to Skip-gram, which generates four separate training pairs from the same position: (fox → quick), (fox → brown), (fox → jumps), and (fox → over). CBOW instead creates one training example where all four context words together predict "fox."

Architecture: Context AveragingLink Copied

CBOW's architecture is similar to Skip-gram but with a crucial difference in the input layer. Instead of taking a single word as input, CBOW takes multiple context words and averages their embeddings.

The network has two weight matrices, identical in purpose to Skip-gram:

1. Embedding matrix $\mathbf{W}$ (size $V \times d$): Maps input words to dense vectors. Each row is the embedding for one vocabulary word. For CBOW, we look up multiple rows (one per context word) and average them.

2. Context matrix $\mathbf{W}'$ (size $d \times V$): Maps the hidden representation to output scores over the vocabulary.

The key difference from Skip-gram is what happens between the embedding lookup and the output layer: CBOW averages the context embeddings, while Skip-gram uses the single center word embedding directly.

Context Word AveragingLink Copied

How should a model combine information from multiple context words? Skip-gram sidesteps this question entirely, processing each context word independently. But CBOW must somehow merge four separate embeddings into a single representation that captures the collective meaning of the context.

The simplest approach is also highly effective: take the average. If each context word embedding captures something about that word's meaning, then their average should capture what these words have in common. When the context words are semantically coherent (as they typically are around a meaningful center word), this average points toward the semantic region where the center word belongs.

This leads us to the core formula of CBOW:

where:

• $\bar{\mathbf{h}}$: the averaged hidden representation (the "centroid" of the context)
• $C$: the number of context words (typically $2m$ for window size $m$)
• $\mathbf{w}_{c_i}$: the embedding vector of the $i$-th context word (row $c_i$ of $\mathbf{W}$)

The averaging operation is simple, just element-wise means across all context embeddings. Yet this simplicity carries important implications for what CBOW can and cannot learn.

Each dimension of the averaged embedding is simply the mean of the corresponding dimensions from the individual word embeddings. Notice how the averaged vector smooths out the individual variations, producing values that lie between the extremes of the component vectors.

Visualizing Context AveragingLink Copied

Let's visualize how averaging combines multiple context vectors into a single representation:

The averaged vector represents the semantic centroid of the context. When context words are semantically coherent (as they typically are around a meaningful center word), this centroid points toward the region of embedding space where the center word should reside.

The CBOW Objective FunctionLink Copied

With the averaged context representation $\bar{\mathbf{h}}$ in hand, CBOW faces a classification problem: which of the $V$ vocabulary words is most likely to appear in the center position? This is where the second weight matrix, $\mathbf{W}'$, comes into play.

From Context to PredictionLink Copied

The intuition behind the prediction step is geometric. Each word in the vocabulary has a corresponding vector in the context matrix $\mathbf{W}'$. To predict the center word, CBOW computes how well the averaged context $\bar{\mathbf{h}}$ aligns with each vocabulary word's context vector. The word whose context vector points most in the same direction as $\bar{\mathbf{h}}$ receives the highest probability.

This alignment is measured by the dot product. For the target word $w_t$, we compute $\mathbf{w}'_t \cdot \bar{\mathbf{h}}$, which yields a large positive value when the vectors point in similar directions. But we need probabilities, not raw scores, so we apply softmax normalization:

where:

• $w_t$: the target center word we're trying to predict
• $\bar{\mathbf{h}}$: the averaged context embedding (computed from the input layer)
• $\mathbf{w}'_t$: the context vector for word $w_t$ (column $t$ of $\mathbf{W}'$)
• $V$: vocabulary size

The softmax denominator sums over all vocabulary words, ensuring the probabilities sum to 1. This is the same formulation as Skip-gram, but with the averaged context vector $\bar{\mathbf{h}}$ replacing the single center word embedding.

From Probability to LossLink Copied

Training requires a loss function that measures prediction quality. The natural choice is cross-entropy loss, which penalizes the model for assigning low probability to the correct center word. Taking the negative logarithm of the probability gives us:

This loss has an intuitive interpretation. The first term, $-\mathbf{w}'_t \cdot \bar{\mathbf{h}}$, wants to maximize the dot product between the context representation and the correct word, pushing them to align in embedding space. The second term, the log-sum-exp, acts as a normalizing penalty that prevents all dot products from growing unboundedly large.

The Corpus ObjectiveLink Copied

For a complete training corpus, we average the loss over all word positions:

where $N$ is the total number of training positions in the corpus and $\mathcal{L}_n$ is the loss at position $n$. Minimizing this objective encourages the model to correctly predict center words from their contexts across the entire corpus.

With random weights, the model has no preference for the correct word. Training adjusts the embeddings so that the averaged context vector produces high probability for the actual center word.

CBOW vs Skip-gram: A Detailed ComparisonLink Copied

While CBOW and Skip-gram share the same embedding matrices, they differ fundamentally in how they use training data.

Training Signal DensityLink Copied

Consider the sentence "The quick brown fox jumps over the lazy dog" with window size 2. At position 3 (word "fox"):

Skip-gram generates 4 training pairs:

• (fox → quick)
• (fox → brown)
• (fox → jumps)
• (fox → over)

Each pair updates the "fox" embedding based on predicting one context word.

CBOW generates 1 training example:

• (quick, brown, jumps, over → fox)

This single example updates all four context word embeddings based on predicting "fox."

Implications for Rare vs. Frequent WordsLink Copied

This difference in training structure has important consequences:

Rare words benefit more from Skip-gram:

• Each occurrence of a rare word in Skip-gram generates multiple training examples (one per context word)
• In CBOW, a rare word appearing as center word creates only one training example
• A rare word appearing in context contributes only a fraction (1/C) of the gradient

Frequent words can benefit from CBOW:

• CBOW's averaging smooths out noisy context patterns
• For frequent words with many training examples, the averaging helps learn stable representations
• The single prediction per position makes training faster

Skip-gram provides proportionally more training signal per word occurrence. For a rare word like "ceremony" (appearing once), Skip-gram creates four training pairs, while CBOW creates only one (plus fractional context contributions). This difference explains why Skip-gram typically produces better representations for rare words.

Training SpeedLink Copied

CBOW is faster to train than Skip-gram for two reasons:

1. Fewer forward/backward passes: CBOW makes one prediction per position; Skip-gram makes $2m$ predictions
2. Shared computation: The context averaging in CBOW reuses embeddings; Skip-gram computes independently

The benchmark confirms the theoretical speedup. CBOW processes each corpus position faster because it makes one prediction rather than multiple predictions. For corpora with billions of words, this speedup translates to hours or even days of saved training time.

Summary of Trade-offsLink Copied

The Forward PassLink Copied

Let's implement the complete CBOW forward pass. Given context words, we compute the probability distribution over vocabulary words:

The untrained model assigns roughly uniform probabilities across vocabulary words, reflecting its lack of learned associations. After training, the model will concentrate probability mass on the actual center word.

Gradient DerivationLink Copied

With the forward pass defined, we now derive the gradients needed to train CBOW. Understanding these gradients reveals why CBOW behaves differently from Skip-gram during training, particularly the crucial insight that context words share the gradient signal.

Recall the loss for a single training example:

Training requires computing how this loss changes with respect to each weight in the model. We'll work backward through the network, starting at the output and flowing gradients back to the input embeddings.

Gradient for the Context Matrix $\mathbf{W}'$Link Copied

The context matrix $\mathbf{W}'$ directly produces the output scores. Each column $\mathbf{w}'_j$ represents word $j$ in the output layer. How should we adjust these vectors to reduce the loss?

Taking the derivative with respect to the output vector $\mathbf{w}'_j$ for any word $j$:

where $\mathbb{1}_{j=t}$ is the indicator function: 1 if $j$ is the target word, 0 otherwise.

The first term pushes the target word's context vector toward the averaged context $\bar{\mathbf{h}}$, since we want high dot product for the correct answer. The second term pulls all words away from $\bar{\mathbf{h}}$ in proportion to their predicted probability. The model is "too confident" about incorrect words if it assigns them high probability.

Recognizing that the second term is simply the softmax probability $P(w_j)$, we can simplify:

This formula has an intuitive interpretation: the gradient is proportional to the "prediction error." For the target word, the error is $P(w_t) - 1$ (negative, so we push toward $\bar{\mathbf{h}}$). For all other words, the error is $P(w_j) - 0 = P(w_j)$ (positive, so we push away from $\bar{\mathbf{h}}$).

In matrix form, we can write this compactly as:

where:

• $\otimes$: the outer product operation
• $\mathbf{p}$: the softmax output probability vector (dimension $V$)
• $\mathbf{y}$: the one-hot target vector (1 at position $t$, 0 elsewhere)

Gradient for the Embedding Matrix $\mathbf{W}$Link Copied

The gradient must flow backward through the averaging operation to reach the input embeddings. This is where CBOW fundamentally differs from Skip-gram.

First, we compute the gradient with respect to the averaged representation $\bar{\mathbf{h}}$. This requires multiplying by the context matrix, which maps from the hidden dimension back to the vocabulary:

This gradient tells us how the loss would change if we nudged $\bar{\mathbf{h}}$ in any direction. But $\bar{\mathbf{h}}$ is not a learnable parameter. It's computed as the average of the context word embeddings, so we must distribute this gradient to the actual parameters: the embedding vectors $\mathbf{w}_{c_1}, \mathbf{w}_{c_2}, \ldots, \mathbf{w}_{c_C}$.

Here's the critical insight: since $\bar{\mathbf{h}} = \frac{1}{C} \sum_{i=1}^{C} \mathbf{w}_{c_i}$, each context word contributes equally to the average. By the chain rule, the gradient splits equally among them:

Each context word embedding receives exactly $1/C$ of the gradient signal. This division has important implications:

1. Diluted learning for rare words: A rare word appearing in a context of four words receives only 25% of the gradient that a center word would receive in Skip-gram.

2. Smoothed updates: The averaging acts as implicit regularization, preventing any single context word from dominating the update.

3. Faster training: Despite the dilution, CBOW processes each corpus position with a single forward-backward pass, making it faster overall.

Complete Implementation with TrainingLink Copied

The mathematical framework is now complete: we know how to compute predictions (forward pass) and how to compute gradients for learning (backward pass). Let's bring these pieces together into a working implementation.

The implementation follows the gradient equations exactly. In the backward pass, notice how the gradient first flows to $\bar{\mathbf{h}}$ through the context matrix, then divides by $C$ before reaching each context word embedding. This is the $1/C$ gradient division we derived earlier.

Training on a Toy CorpusLink Copied

The loss reduction indicates the model has learned to predict center words from their context. The initial loss reflects random guessing across the vocabulary, while the final loss shows the model assigns higher probability to actual center words.

Examining Learned EmbeddingsLink Copied

The model has learned that words appearing in similar contexts have similar embeddings. Words from the same semantic category cluster together in embedding space, demonstrating that CBOW successfully captures distributional semantics even on this small corpus.

When to Choose CBOWLink Copied

Given the trade-offs between CBOW and Skip-gram, when should you reach for CBOW? The model excels in specific scenarios:

Large corpora with common words: When you have billions of words and care most about representing frequent vocabulary, CBOW's faster training and averaging-based smoothing are advantageous.

Time-constrained training: If computational resources are limited, CBOW's 3-4x speedup over Skip-gram can make training feasible where it otherwise wouldn't be.

Syntactic tasks: Some research suggests CBOW performs slightly better on syntactic analogy tasks (e.g., "big : bigger :: small : ?"), possibly because averaging captures grammatical patterns effectively.

Downstream averaging: If your application averages word embeddings to create sentence or document representations, CBOW's training objective aligns naturally with this use case.

Limitations of Context AveragingLink Copied

While CBOW's averaging approach enables faster training and smoother representations, the bag-of-words assumption introduces inherent limitations that affect what the model can learn:

Order insensitivity: "dog bites man" and "man bites dog" produce identical averaged context representations, even though they mean very different things.

Dilution of rare context words: A rare but informative context word contributes only $1/C$ of the gradient, potentially getting drowned out by more common words in the same context window.

Position blindness: A word immediately adjacent to the target is treated the same as a word at the edge of the window. Some extensions weight by distance to address this.

Position-Weighted VariantsLink Copied

The position blindness limitation motivates an extension to standard CBOW. Some implementations weight context words by their distance from the target, giving nearby words more influence on the final representation:

where:

• $\alpha_i$: the weight for the $i$-th context word (decreases with distance from center)
• $\mathbf{w}_{c_i}$: the embedding vector of the $i$-th context word
• $C$: the number of context words

Common weighting schemes include:

• Linear decay: $\alpha_i = 1 - \frac{|d_i|}{m+1}$ where $d_i$ is the position offset from the center word and $m$ is the window size
• Inverse distance: $\alpha_i = \frac{1}{|d_i|}$ where $d_i$ is the position offset
• Exponential decay: $\alpha_i = \exp(-\lambda |d_i|)$ where $\lambda$ controls the decay rate

Implementing Position WeightingLink Copied

Let's implement a weighted averaging function that supports different distance-based weighting schemes:

The different weighting schemes produce distinct averaged embeddings. With inverse distance weighting, the words at positions -1 and +1 (closest to the center) contribute more to the final representation than those at -2 and +2. This can improve embedding quality when nearby context words carry more semantic relevance than distant ones.

The Softmax Bottleneck (Again)Link Copied

Like Skip-gram, CBOW faces the softmax computational bottleneck. Each training step requires computing:

The denominator sums over all $V$ vocabulary words. For a vocabulary of 100,000 words, this means computing 100,000 dot products and exponentials per training step, a significant computational burden. The same approximation techniques that accelerate Skip-gram also apply to CBOW:

• Negative sampling: Sample a small number of negative examples instead of computing the full softmax
• Hierarchical softmax: Use a binary tree structure to reduce complexity from $O(V)$ to $O(\log V)$

We'll cover these techniques in detail in subsequent chapters.

Key TakeawaysLink Copied

CBOW and Skip-gram are complementary approaches to learning word embeddings from context:

• Architectural difference: CBOW averages context embeddings to predict the center word; Skip-gram uses the center word to predict each context word
• Training dynamics: CBOW creates one training example per position, Skip-gram creates $2m$ examples. CBOW is faster but provides less training signal for rare words
• Context averaging: The "bag of words" assumption treats context as an unordered set, losing word order information but capturing overall semantic context
• Gradient distribution: Each context word receives $1/C$ of the gradient, diluting the signal for any single word. This explains why rare words benefit more from Skip-gram
• Practical trade-offs: Choose CBOW when training speed matters and vocabulary is dominated by frequent words. Choose Skip-gram when rare word quality matters

The next chapter explores negative sampling, which accelerates both CBOW and Skip-gram by replacing the expensive softmax with a simpler binary classification objective.

Key ParametersLink Copied

When training CBOW models, several hyperparameters impact embedding quality:

embedding_dim (typical range: 50-300): The dimensionality of word vectors. Lower values (50-100) provide faster training and smaller memory footprint, sufficient for many tasks. Higher values (200-300) capture more nuanced relationships but require more data. A common choice is 100-200 for most applications.

window_size (typical range: 2-10): Number of context words on each side of the center word. Small windows (2-3) emphasize syntactic relationships, while large windows (5-10) capture broader topical similarity. A common choice is 5 for balanced representations.

min_count (typical range: 1-100): Minimum word frequency to include in vocabulary. Lower values include rare words but with potentially unreliable embeddings. Higher values produce more robust embeddings for included words. A common choice is 5-10 for large corpora.

learning_rate (typical range: 0.01-0.1): Step size for gradient descent updates. Higher values enable faster convergence but may overshoot. Lower values are more stable but slower. A common choice is 0.025-0.05 with linear decay.

epochs (typical range: 1-20): Number of passes through the training corpus. Fewer epochs mean faster training but may underfit. More epochs provide better convergence, with diminishing returns after 5-10. A common choice is 5 epochs for large corpora.

negative_samples (when using negative sampling, typical range: 5-20): Number of negative examples per positive example. Fewer negatives (5) enable faster training. More negatives (15-20) provide better discrimination. A common choice is 5-10 for large corpora.

SummaryLink Copied

CBOW learns word embeddings by predicting center words from their surrounding context. The model averages context word embeddings into a single representation, then uses this averaged vector to predict the center word via softmax. This averaging operation, the defining feature of CBOW, treats context as an unordered "bag of words," ignoring position information but capturing overall semantic context effectively.

Compared to Skip-gram, CBOW trains 3-4x faster due to making a single prediction per position rather than one per context word. However, the gradient division inherent in averaging means each context word receives only a fraction of the training signal, which can weaken representations for rare words. CBOW tends to perform better on frequent words and syntactic tasks, while Skip-gram excels at rare word representation. The choice between them depends on your corpus characteristics and downstream application requirements.

Both CBOW and Skip-gram share a common computational challenge: the softmax normalization requires summing over the entire vocabulary for each training step. The next chapter introduces negative sampling, an approximation technique that dramatically reduces this cost while maintaining embedding quality.

QuizLink Copied

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the CBOW model and its differences from Skip-gram.