CBOW Model: Learning Word Embeddings by Predicting Center Words

In the previous chapter, we explored Skip-gram, which learns word embeddings by predicting context words from a center word. The Continuous Bag of Words (CBOW) model takes the opposite approach: given the surrounding context words, predict the center word. This architectural inversion leads to different learning dynamics and computational trade-offs.

CBOW was introduced alongside Skip-gram in the original Word2Vec paper by Mikolov et al. (2013). While Skip-gram treats each context position independently, CBOW averages context word embeddings together to make a single prediction per training example. This design choice has practical implications: CBOW trains faster than Skip-gram and performs better on frequent words, while Skip-gram excels at representing rare words.

This chapter covers the CBOW architecture from the ground up. We'll work through the mathematics, implement the model from scratch, and understand when to choose CBOW over Skip-gram.

Imagine reading a sentence with a word missing: "The quick brown ___ jumps over the lazy dog." Given the surrounding context ("quick," "brown," "jumps," "over"), you can likely guess the missing word is "fox." This fill-in-the-blank task is exactly what CBOW learns to do.

The name "Continuous Bag of Words" comes from two properties:

1. Continuous: The model uses continuous-valued vectors (embeddings) rather than discrete word representations
2. Bag of Words: The context words are treated as an unordered set, averaging their embeddings together regardless of position

This single training example captures the essence of CBOW: given the surrounding words, predict what goes in the middle. Compare this to Skip-gram, which generates four separate training pairs from the same position: (fox → quick), (fox → brown), (fox → jumps), and (fox → over). CBOW instead creates one training example where all four context words together predict "fox."

CBOW's architecture is similar to Skip-gram but with a crucial difference in the input layer. Instead of taking a single word as input, CBOW takes multiple context words and averages their embeddings.

The network has two weight matrices, identical in purpose to Skip-gram:

1. Embedding matrix $\mathbf{W}$ (size $V \times d$): Maps input words to dense vectors. Each row is the embedding for one vocabulary word. For CBOW, we look up multiple rows (one per context word) and average them.

2. Context matrix $\mathbf{W}'$ (size $d \times V$): Maps the hidden representation to output scores over the vocabulary.

The key difference from Skip-gram is what happens between the embedding lookup and the output layer: CBOW averages the context embeddings, while Skip-gram uses the single center word embedding directly.

How should a model combine information from multiple context words? Skip-gram sidesteps this question entirely, processing each context word independently. But CBOW must somehow merge four separate embeddings into a single representation that captures the collective meaning of the context.

The simplest approach is also highly effective: take the average. If each context word embedding captures something about that word's meaning, then their average should capture what these words have in common. When the context words are semantically coherent (as they typically are around a meaningful center word), this average points toward the semantic region where the center word belongs.

This leads us to the core formula of CBOW. Given $C$ context words, the model computes their centroid in embedding space by taking the element-wise mean of their embedding vectors:

where:

• $\bar{\mathbf{h}}$: the averaged hidden representation (a $d$-dimensional vector representing the "centroid" of the context)
• $C$: the number of context words (typically $2m$ for a symmetric window of size $m$ on each side)
• $\mathbf{w}_{c_i}$: the $d$-dimensional embedding vector of the $i$-th context word (row $c_i$ of the embedding matrix $\mathbf{W}$)
• $\sum_{i=1}^{C}$: summation over all $C$ context word embeddings

The averaging operation is simple: for each dimension $j$ of the embedding, $\bar{h}_j = \frac{1}{C}(w_{c_1,j} + w_{c_2,j} + \cdots + w_{c_C,j})$. This element-wise mean produces values that lie between the extremes of the component vectors, smoothing out individual word idiosyncrasies while preserving shared semantic content.

This simplicity carries important implications for what CBOW can and cannot learn.

Each dimension of the averaged embedding is simply the mean of the corresponding dimensions from the individual word embeddings. Notice how the averaged vector smooths out the individual variations, producing values that lie between the extremes of the component vectors.

Let's visualize how averaging combines multiple context vectors into a single representation:

The averaged vector represents the semantic centroid of the context. When context words are semantically coherent (as they typically are around a meaningful center word), this centroid points toward the region of embedding space where the center word should reside.

With the averaged context representation $\bar{\mathbf{h}}$ in hand, CBOW faces a classification problem: which of the $V$ vocabulary words is most likely to appear in the center position? This is where the second weight matrix, $\mathbf{W}'$, comes into play.

The intuition behind the prediction step is geometric. Each word in the vocabulary has a corresponding vector in the context matrix $\mathbf{W}'$. To predict the center word, CBOW computes how well the averaged context $\bar{\mathbf{h}}$ aligns with each vocabulary word's context vector. The word whose context vector points most in the same direction as $\bar{\mathbf{h}}$ receives the highest probability.

This alignment is measured by the dot product. For the target word $w_t$, we compute $\mathbf{w}'_t \cdot \bar{\mathbf{h}}$, which yields a large positive value when the vectors point in similar directions. But we need probabilities, not raw scores, so we apply softmax normalization.

The softmax function converts a vector of real-valued scores into a probability distribution. Applied to our word prediction task, it computes:

where:

• $P(w_t | w_{c_1}, \ldots, w_{c_C})$: the probability of word $w_t$ being the center word, given context words $w_{c_1}, \ldots, w_{c_C}$
• $w_t$: the target center word we're trying to predict
• $\bar{\mathbf{h}}$: the averaged context embedding (computed from the input layer)
• $\mathbf{w}'_t$: the context vector for word $w_t$ (column $t$ of $\mathbf{W}'$)
• $\exp(\mathbf{w}'_t \cdot \bar{\mathbf{h}})$: the exponential of the dot product, ensuring the numerator is always positive
• $\sum_{k=1}^{V} \exp(\mathbf{w}'_k \cdot \bar{\mathbf{h}})$: the normalization constant, summing over all $V$ vocabulary words to ensure probabilities sum to 1
• $V$: vocabulary size (total number of words in the vocabulary)

The exponential function serves two purposes: (1) it maps any real-valued dot product to a positive number, ensuring valid probabilities, and (2) it amplifies differences between scores, so words with higher dot products receive exponentially higher probability mass. The denominator normalizes these values so they sum to 1 across the vocabulary.

This is the same formulation as Skip-gram, but with the averaged context vector $\bar{\mathbf{h}}$ replacing the single center word embedding.

Training requires a loss function that measures prediction quality. The natural choice is cross-entropy loss, which penalizes the model for assigning low probability to the correct center word.

Starting from the probability formula, we take the negative logarithm to obtain the loss:

Step 1: Begin with the softmax probability for the target word:

Step 2: Apply the negative log to convert to a loss (lower probability → higher loss):

Step 3: Apply the logarithm quotient rule, $\log(a/b) = \log(a) - \log(b)$:

Step 4: Simplify using $\log \exp(x) = x$:

where:

• $\mathcal{L}$: the cross-entropy loss for a single training example (a non-negative scalar)
• $\mathbf{w}'_t \cdot \bar{\mathbf{h}}$: the dot product (alignment score) between the target word's context vector and the averaged context embedding
• $\log \sum_{k=1}^{V} \exp(\mathbf{w}'_k \cdot \bar{\mathbf{h}})$: the "log-sum-exp" normalization term

This loss has an intuitive interpretation. The first term, $-\mathbf{w}'_t \cdot \bar{\mathbf{h}}$, wants to maximize the dot product between the context representation and the correct word, pushing them to align in embedding space. The second term, the log-sum-exp, acts as a normalizing penalty that prevents all dot products from growing unboundedly large. Without this term, the model could trivially reduce loss by scaling all vectors to infinity.

The loss $\mathcal{L}$ applies to a single training example. For a complete training corpus, we aggregate the losses over all word positions by computing their average:

where:

• $J$: the corpus-level objective function (average loss across the entire corpus)
• $N$: the total number of training positions in the corpus (each position where we extract a center word and its context)
• $\mathcal{L}_n$: the cross-entropy loss at position $n$ (computed using the formula derived above)
• $\sum_{n=1}^{N}$: summation over all training positions

Minimizing this objective encourages the model to correctly predict center words from their contexts across the entire corpus. During training, we use stochastic gradient descent to update the embedding matrices $\mathbf{W}$ and $\mathbf{W}'$ to reduce $J$.

With random weights, the model has no preference for the correct word. Training adjusts the embeddings so that the averaged context vector produces high probability for the actual center word.

While CBOW and Skip-gram share the same embedding matrices, they differ fundamentally in how they use training data.

Consider the sentence "The quick brown fox jumps over the lazy dog" with window size 2. At position 3 (word "fox"):

Skip-gram generates 4 training pairs:

• (fox → quick)
• (fox → brown)
• (fox → jumps)
• (fox → over)

Each pair updates the "fox" embedding based on predicting one context word.

CBOW generates 1 training example:

• (quick, brown, jumps, over → fox)

This single example updates all four context word embeddings based on predicting "fox."

This difference in training structure has important consequences:

Rare words benefit more from Skip-gram:

• Each occurrence of a rare word in Skip-gram generates multiple training examples (one per context word)
• In CBOW, a rare word appearing as center word creates only one training example
• A rare word appearing in context contributes only a fraction (1/C) of the gradient

Frequent words can benefit from CBOW:

• CBOW's averaging smooths out noisy context patterns
• For frequent words with many training examples, the averaging helps learn stable representations
• The single prediction per position makes training faster

Table training-signal quantifies how Skip-gram provides proportionally more training signal per word occurrence.

CBOW is faster to train than Skip-gram for two reasons:

1. Fewer forward/backward passes: CBOW makes one prediction per position; Skip-gram makes $2m$ predictions
2. Shared computation: The context averaging in CBOW reuses embeddings; Skip-gram computes independently

The benchmark confirms the theoretical speedup. CBOW processes each corpus position faster because it makes one prediction rather than multiple predictions. For corpora with billions of words, this speedup translates to hours or even days of saved training time.

Let's implement the complete CBOW forward pass. Given context words, we compute the probability distribution over vocabulary words:

The untrained model assigns roughly uniform probabilities across vocabulary words, reflecting its lack of learned associations. After training, the model will concentrate probability mass on the actual center word.

With the forward pass defined, we now derive the gradients needed to train CBOW. Understanding these gradients reveals why CBOW behaves differently from Skip-gram during training, particularly the crucial insight that context words share the gradient signal.

Recall the loss for a single training example:

Training requires computing how this loss changes with respect to each weight in the model. We'll work backward through the network, starting at the output and flowing gradients back to the input embeddings.

The context matrix $\mathbf{W}'$ directly produces the output scores. Each column $\mathbf{w}'_j$ represents word $j$ in the output layer. To train the model, we need to compute how the loss changes when we adjust these vectors.

Step 1: Recall that the loss depends on $\mathbf{w}'_j$ through the term $\mathbf{w}'_j \cdot \bar{\mathbf{h}}$. For the target word $t$, this appears in the numerator; for all words, it appears in the denominator.

Step 2: Taking the partial derivative of the loss with respect to $\mathbf{w}'_j$ for any word $j$:

where:

• $\frac{\partial \mathcal{L}}{\partial \mathbf{w}'_j}$: the gradient of the loss with respect to the context vector for word $j$ (a $d$-dimensional vector)
• $\mathbb{1}_{j=t}$: the indicator function, equal to 1 if $j = t$ (the target word), and 0 otherwise
• $\bar{\mathbf{h}}$: the averaged context embedding
• The fraction is the softmax probability $P(w_j)$

The first term, $-\mathbb{1}_{j=t} \cdot \bar{\mathbf{h}}$, only appears for the target word. It pushes $\mathbf{w}'_t$ toward $\bar{\mathbf{h}}$ to increase the dot product (and thus the probability) for the correct answer. The second term pulls all words' context vectors away from $\bar{\mathbf{h}}$ in proportion to their predicted probability.

Step 3: Recognizing that the fraction is simply $P(w_j)$, we can simplify:

This elegant formula shows the gradient is proportional to the "prediction error":

• For the target word ($j = t$): The gradient is $(P(w_t) - 1) \cdot \bar{\mathbf{h}}$. Since $P(w_t) < 1$, this is negative, meaning we increase $\mathbf{w}'_t \cdot \bar{\mathbf{h}}$ (push the vectors to align).
• For all other words ($j \neq t$): The gradient is $P(w_j) \cdot \bar{\mathbf{h}}$. This is positive, meaning we decrease $\mathbf{w}'_j \cdot \bar{\mathbf{h}}$ (push the vectors apart).

Step 4: In matrix form, we can express this compactly using the outer product:

where:

• $\frac{\partial \mathcal{L}}{\partial \mathbf{W}'}$: the gradient matrix (size $d \times V$)
• $\bar{\mathbf{h}}$: the averaged context embedding (size $d \times 1$)
• $\otimes$: the outer product operation, producing a $d \times V$ matrix from vectors of size $d$ and $V$
• $\mathbf{p}$: the softmax output probability vector (size $V \times 1$, with entries $P(w_j)$ for each word)
• $\mathbf{y}$: the one-hot target vector (size $V \times 1$, with 1 at position $t$ and 0 elsewhere)
• $(\mathbf{p} - \mathbf{y})$: the error vector, measuring how far each predicted probability is from the target

The gradient must flow backward through the averaging operation to reach the input embeddings. This is where CBOW fundamentally differs from Skip-gram.

Step 1: Compute the gradient with respect to the averaged representation $\bar{\mathbf{h}}$. Using the chain rule, this requires multiplying by the context matrix:

where:

• $\frac{\partial \mathcal{L}}{\partial \bar{\mathbf{h}}}$: the gradient of the loss with respect to the averaged context vector (a $d$-dimensional vector)
• $\mathbf{W}'$: the context matrix (size $d \times V$)
• $(\mathbf{p} - \mathbf{y})$: the error vector (size $V \times 1$)
• The matrix-vector product $\mathbf{W}' (\mathbf{p} - \mathbf{y})$ produces a $d$-dimensional vector

This gradient tells us how the loss would change if we nudged $\bar{\mathbf{h}}$ in any direction. But $\bar{\mathbf{h}}$ is not a learnable parameter; it's computed as the average of the context word embeddings. We must therefore distribute this gradient to the actual parameters: the embedding vectors $\mathbf{w}_{c_1}, \mathbf{w}_{c_2}, \ldots, \mathbf{w}_{c_C}$.

Step 2: Apply the chain rule to propagate the gradient through the averaging operation. Recall that:

Taking the partial derivative of $\bar{\mathbf{h}}$ with respect to any context word embedding $\mathbf{w}_{c_i}$:

Each context word contributes exactly $1/C$ to the average, so each receives $1/C$ of the gradient.

Step 3: Apply the chain rule to get the gradient for each context word embedding:

where:

• $\frac{\partial \mathcal{L}}{\partial \mathbf{w}_{c_i}}$: the gradient for the $i$-th context word embedding (a $d$-dimensional vector)
• $\frac{1}{C}$: the scaling factor from the averaging operation
• $\frac{\partial \mathcal{L}}{\partial \bar{\mathbf{h}}}$: the gradient at the averaged representation (computed in Step 1)

Each context word embedding receives exactly $\frac{1}{C}$ of the gradient signal. This division has important implications:

1. Diluted learning for rare words: A rare word appearing in a context of four words receives only 25% of the gradient that a center word would receive in Skip-gram.

2. Smoothed updates: The averaging acts as implicit regularization, preventing any single context word from dominating the update.

3. Faster training: Despite the dilution, CBOW processes each corpus position with a single forward-backward pass, making it faster overall.

The mathematical framework is now complete: we know how to compute predictions (forward pass) and how to compute gradients for learning (backward pass). Let's bring these pieces together into a working implementation.

The implementation follows the gradient equations exactly. In the backward pass, notice how the gradient first flows to $\bar{\mathbf{h}}$ through the context matrix, then divides by $C$ before reaching each context word embedding. This is the $1/C$ gradient division we derived earlier.

The loss reduction indicates the model has learned to predict center words from their context. The initial loss reflects random guessing across the vocabulary, while the final loss shows the model assigns higher probability to actual center words.

The model has learned that words appearing in similar contexts have similar embeddings. Words from the same semantic category cluster together in embedding space, demonstrating that CBOW successfully captures distributional semantics even on this small corpus.

Given the trade-offs between CBOW and Skip-gram, when should you reach for CBOW? The model excels in specific scenarios:

• Large corpora with common words: When you have billions of words and care most about representing frequent vocabulary, CBOW's faster training and averaging-based smoothing are advantageous.
• Time-constrained training: If computational resources are limited, CBOW's 3-4x speedup over Skip-gram can make training feasible where it otherwise wouldn't be.
• Syntactic tasks: Some research suggests CBOW performs slightly better on syntactic analogy tasks (e.g., "big : bigger :: small : ?"), possibly because averaging captures grammatical patterns effectively.
• Downstream averaging: If your application averages word embeddings to create sentence or document representations, CBOW's training objective aligns naturally with this use case.

While CBOW's averaging approach enables faster training and smoother representations, the bag-of-words assumption introduces inherent limitations that affect what the model can learn:

• Order insensitivity: "dog bites man" and "man bites dog" produce identical averaged context representations, even though they mean very different things.
• Dilution of rare context words: A rare but informative context word contributes only $1/C$ of the gradient, potentially getting drowned out by more common words in the same context window.
• Position blindness: A word immediately adjacent to the target is treated the same as a word at the edge of the window. Some extensions weight by distance to address this.

The position blindness limitation motivates an extension to standard CBOW. In standard CBOW, all context words contribute equally to the averaged representation, regardless of their distance from the center word. However, words immediately adjacent to the target often carry more semantic relevance than words at the edge of the window.

Some implementations address this by weighting context words based on their distance from the target. Instead of a simple average, we compute a weighted average:

where:

• $\bar{\mathbf{h}}$: the weighted averaged hidden representation (a $d$-dimensional vector)
• $C$: the number of context words in the window
• $\alpha_i$: the weight for the $i$-th context word (typically decreasing with distance from center)
• $\mathbf{w}_{c_i}$: the embedding vector of the $i$-th context word
• $\sum_{i=1}^{C} \alpha_i$: the sum of all weights, used to normalize the weighted sum

The numerator computes a weighted sum of the context embeddings, and the denominator normalizes by the total weight. When all $\alpha_i = 1$, this reduces to the standard averaging formula.

Common weighting schemes define $\alpha_i$ based on the position offset $d_i$ (the signed distance from the center word):

• Linear decay: $\alpha_i = 1 - \frac{|d_i|}{m+1}$, where $|d_i|$ is the absolute distance from the center word and $m$ is the window size. This gives weight 1 to the closest words and linearly decreases to 0 at distance $m+1$.
• Inverse distance: $\alpha_i = \frac{1}{|d_i|}$, where $|d_i|$ is the absolute distance. Words at position $\pm 1$ receive weight 1, words at $\pm 2$ receive weight 0.5, and so on.
• Exponential decay: $\alpha_i = \exp(-\lambda |d_i|)$, where $\lambda > 0$ is a hyperparameter controlling the decay rate. Larger $\lambda$ values concentrate more weight on nearby words.

Let's implement a weighted averaging function that supports different distance-based weighting schemes:

The different weighting schemes produce distinct averaged embeddings. With inverse distance weighting, the words at positions -1 and +1 (closest to the center) contribute more to the final representation than those at -2 and +2. This can improve embedding quality when nearby context words carry more semantic relevance than distant ones.

Like Skip-gram, CBOW faces the softmax computational bottleneck. Recall that each training step requires computing the probability for the target word:

where:

• $P(w_t | \text{context})$: the probability of the target word given the context
• $\mathbf{w}'_t$: the context vector for the target word (column $t$ of $\mathbf{W}'$)
• $\bar{\mathbf{h}}$: the averaged context embedding
• $V$: the vocabulary size
• $\sum_{k=1}^{V}$: the expensive summation over all vocabulary words

The computational bottleneck lies in the denominator. For each training step, we must:

1. Compute $V$ dot products: $\mathbf{w}'_k \cdot \bar{\mathbf{h}}$ for every word $k$ in the vocabulary
2. Compute $V$ exponentials: $\exp(\mathbf{w}'_k \cdot \bar{\mathbf{h}})$ for each word
3. Sum all $V$ exponentials to get the normalization constant

For a vocabulary of 100,000 words, this means 100,000 dot products and exponentials per training step, a significant computational burden that scales linearly with vocabulary size, $O(V)$.

The same approximation techniques that accelerate Skip-gram also apply to CBOW:

• Negative sampling: Sample a small number of negative examples instead of computing the full softmax, reducing per-step complexity from $O(V)$ to $O(k)$ where $k \ll V$
• Hierarchical softmax: Use a binary tree structure to reduce complexity from $O(V)$ to $O(\log V)$

We'll cover these techniques in detail in subsequent chapters.

CBOW and Skip-gram are complementary approaches to learning word embeddings from context:

• Architectural difference: CBOW averages context embeddings to predict the center word; Skip-gram uses the center word to predict each context word
• Training dynamics: CBOW creates one training example per position, Skip-gram creates $2m$ examples. CBOW is faster but provides less training signal for rare words
• Context averaging: The "bag of words" assumption treats context as an unordered set, losing word order information but capturing overall semantic context
• Gradient distribution: Each context word receives $1/C$ of the gradient, diluting the signal for any single word. This explains why rare words benefit more from Skip-gram
• Practical trade-offs: Choose CBOW when training speed matters and vocabulary is dominated by frequent words. Choose Skip-gram when rare word quality matters