CBOW Model: Architecture, Gradients, Word Embedding Training

In the previous chapter, we explored the Skip-gram model, which learns word embeddings by training a neural network to predict surrounding context words from a single center word. The Continuous Bag of Words (CBOW) model reverses this relationship: given several surrounding context words, predict the center word that belongs in the middle. This seemingly small inversion produces a model with different strengths, training dynamics, and practical applications.

Both architectures were introduced by Mikolov et al. in the 2013 Word2Vec paper. While Skip-gram excels at capturing representations for rare words, CBOW trains faster, scales better to large corpora, and produces stronger embeddings for frequent words. Understanding both models gives you a complete picture of the Word2Vec framework and helps you choose the right tool for a given task.

This chapter covers the CBOW architecture from the ground up. We build intuition for why averaging context embeddings works, work through the mathematics of the forward pass and objective function, derive the gradients used in training, implement a working CBOW model from scratch, and compare the two architectures in depth.

Think about the fill-in-the-blank task: "The quick brown ___ jumps over the lazy dog." Given the surrounding words, you can immediately guess "fox." Your brain combines information from all the context words together and uses the collective signal to identify what fits in the middle. CBOW does the same thing computationally.

The name "Continuous Bag of Words" carries a specific meaning. "Continuous" refers to the use of continuous-valued dense vectors rather than discrete one-hot encodings. "Bag of Words" signals that the model treats context as an unordered collection: it does not distinguish between a word appearing two positions to the left and one appearing one position to the right. Both are simply "context words." This positional indifference is a deliberate design choice that simplifies the model and reduces the number of parameters needed.

This design philosophy connects directly to the distributional hypothesis that underlies all word embedding methods. The hypothesis, sometimes paraphrased as "you shall know a word by the company it keeps," holds that words with similar meanings appear in similar contexts. CBOW operationalizes this idea by forcing the model to reconstruct a word from a compressed representation of its neighborhood. If the model learns to do this well, the embedding of each context word must carry information about the kinds of targets it surrounds. After training, those embeddings capture distributional similarity in a dense, continuous vector space.

The key question is: why does averaging the context embeddings work? At first glance, it seems aggressive to collapse four distinct word vectors into a single centroid. The answer lies in how the training signal accumulates across examples. Consider that "fox" appears surrounded by words like "quick," "brown," "runs," "clever," and "jumped" across thousands of sentences in a large corpus. Each training example shifts the embeddings of those context words slightly toward a region of the embedding space associated with fox-like targets. After many such updates, the centroid of those context words consistently points toward the region where "fox" lives. The averaging operation, far from losing information, acts as a kind of implicit regularization: it forces the model to find embeddings that work well collectively rather than relying on any single context word.

This single training example contains one prediction task using four context words. Compare this to Skip-gram: for the same sentence position, Skip-gram would generate four separate training pairs: (fox, quick), (fox, brown), (fox, jumps), and (fox, over). CBOW collapses these into one example. This difference in data efficiency affects training speed in a fundamental way, which we discuss in depth later.

A training corpus is converted into CBOW examples by sliding a window over every position in every sentence. For each position, the center word becomes the target and the words within the window on each side become the context.

The window-based approach encodes an important assumption: words that are closer in a sentence are more relevant to predicting the center word than words that are far away. The window size controls how wide the context is. A small window (size 1 or 2) captures tight syntactic relationships: adjectives next to nouns, verbs near their subjects. A larger window (size 5 or more) captures broader topical co-occurrence: words that appear in the same paragraph because they relate to the same subject. The optimal window size depends on what you want your embeddings to capture.

Notice that positions near the edges of the sentence have fewer context words. The model handles this naturally: those positions contribute fewer context embeddings to average, but the averaging still works. Words at the sentence edges are neither ignored nor given special treatment. The averaging simply operates over whatever context words are available, and the model learns to work with variable-size contexts.

This flexibility is a practical advantage. Unlike models that require fixed-length inputs, CBOW naturally accommodates the varying context sizes that arise at sentence boundaries. There is no padding required and no masking mechanism: the forward pass simply averages the embeddings of however many context words are present.

CBOW shares its overall structure with Skip-gram but adds a critical averaging step between the embedding lookup and the output layer. Understanding this step is essential to understanding both what CBOW can learn and why it makes the design choices it does.

The architecture has three components: an input embedding matrix, a context averaging operation, and an output projection matrix. These three pieces work in sequence to convert a set of context word identifiers into a probability distribution over the vocabulary, and backpropagation through this sequence adjusts the parameters so that the correct center word receives the highest probability.

The two weight matrices in CBOW mirror those in Skip-gram:

• Embedding matrix $\mathbf{W}$ (size $V \times d$): Each row is the $d$-dimensional embedding for one vocabulary word. When CBOW receives context words, it looks up each context word's row and averages them.
• Output matrix $\mathbf{W}'$ (size $d \times V$): Each column corresponds to a vocabulary word's "context representation." This matrix converts the averaged embedding into a score vector over the full vocabulary.

These two matrices serve distinct roles during training even though both are learned simultaneously. The embedding matrix $\mathbf{W}$ is the primary product: these are the word vectors you use after training. The output matrix $\mathbf{W}'$ is more of a scaffolding: it provides the contrastive signal needed to push word embeddings apart during training. Some practitioners use $\mathbf{W}'$ as an alternative embedding matrix (averaging the two is sometimes beneficial), but in most applications only $\mathbf{W}$ is retained.

You might wonder why CBOW has no activation function between the embedding lookup and the output layer. Skip-gram also omits nonlinearities. This is a deliberate choice, and it carries practical effects.

Nonlinearities like ReLU or sigmoid introduce computational cost and complicate backpropagation. More importantly, they change the geometry of the embedding space. A linear architecture means the learned embeddings live in a space where vector arithmetic is meaningful. The famous Word2Vec property that king - man + woman = queen holds precisely because the embedding space is linear. Adding a nonlinearity would destroy this arithmetic structure.

There is also a theoretical argument from information theory. The task of predicting a word from its context is a self-supervised objective that forces the model to compress distributional information into a fixed-size vector. The compression happens through the bottleneck of the embedding dimension $d$, not through a nonlinearity. As long as $d$ is much smaller than $V$, the model must find a compact representation of the word's distributional properties, and that representation turns out to be geometrically meaningful.

The averaging step is CBOW's defining operation. How should the model combine information from multiple context words into a single vector? CBOW takes the simplest possible answer: element-wise mean. If the context words "quick," "brown," "jumps," and "over" each contribute a $d$-dimensional embedding, the averaged representation is the vector whose $j$-th element is the mean of all four embeddings' $j$-th elements.

More formally, given $C$ context words with indices $c_1, c_2, \ldots, c_C$ in the vocabulary, the hidden representation is:

where:

• $\bar{\mathbf{h}}$: the averaged hidden representation, a $d$-dimensional vector representing the "centroid" of the context in embedding space
• $C$: the number of context words (typically $2m$ for a window of size $m$, but fewer at sentence boundaries)
• $\mathbf{w}_{c_i}$: the embedding of the $i$-th context word, the row at index $c_i$ in the embedding matrix $\mathbf{W}$

This averaging has an important geometric interpretation. Each embedding vector corresponds to a point in $d$-dimensional space. The average corresponds to the centroid of those points. If the context words are semantically coherent (which they tend to be in natural language), their centroid points toward the region of the embedding space where the center word should live.

To see why this works, consider what the model must learn for the centroid to be useful. Each context word embedding must encode the word's own meaning and information about what words typically appear near it. After training, the centroid of "quick," "brown," "jumps," and "over" should point toward the region where nouns like "fox" live, because that is the region the model was repeatedly trained to predict from such context centroids.

There is an alternative design choice worth understanding: instead of averaging, you could sum the context embeddings. The sum is mathematically equivalent to the average (they differ only by a constant scaling factor), but the average has a practical advantage: the magnitude of the hidden representation $\bar{\mathbf{h}}$ does not grow with the number of context words. This stability is helpful for maintaining consistent gradient magnitudes during training regardless of window size or sentence boundary effects.

The heatmap makes the averaging operation concrete. Each column of the upper four rows contains the embedding values for one dimension. The bottom row shows how the average smooths these values, placing the centroid between the extremes. In this random-initialization example, the values carry no semantic meaning yet, but the same averaging process applies once the embeddings have been trained. After training, the average of "quick," "brown," "jumps," and "over" will be a vector that the output matrix can reliably decode as "fox."

It is worth dwelling on the geometry a little longer because it explains both the strengths and the limitations of CBOW. In $d$-dimensional embedding space, each word occupies a point. The average of $C$ points is the centroid, the center of mass of the configuration. When the $C$ context words form a tight cluster (because they are semantically similar or syntactically related), the centroid is a stable, representative point. When the context words are semantically diverse or scattered across the embedding space, the centroid can end up in an unusual region, away from any natural cluster.

This geometry explains CBOW.s stability on frequent words. Frequent words appear in many different contexts. Over many training examples, the model encounters "fox" predicted from contexts like ["quick", "brown", "jumps", "over"], ["cunning", "red", "ran", "through"], ["the", "chased", "by", "hound"], and many more. Each of these averages lands near the "fox" region of the space, and the gradients reinforce the embedding accordingly. The signal is consistent and strong.

Rare words, by contrast, may appear in only a handful of contexts. If "dhole" (a type of wild canid) appears in contexts mostly involving predation and forests, its averaged context vectors may cluster in a region that also corresponds to many other animals. The signal is weaker, and neighboring words may not be reliable discriminators. This is why Skip-gram, which trains on each context word individually, tends to outperform CBOW on rare vocabulary.

With the averaged context representation in hand, CBOW needs to assign high probability to the correct center word and low probability to all other vocabulary words. This is a multiclass classification problem across the entire vocabulary.

After computing $\bar{\mathbf{h}}$, the model computes a score for every vocabulary word by multiplying $\bar{\mathbf{h}}$ against the output matrix $\mathbf{W}'$:

where:

• $\mathbf{z}$: a vector of raw scores (logits), one per vocabulary word, shape $V \times 1$
• $\mathbf{W}'$: the output weight matrix of shape $d \times V$
• $\bar{\mathbf{h}}$: the averaged context embedding of shape $d \times 1$

Each element $z_w$ of this score vector measures how well the averaged context aligns with word $w$'s output representation. The dot product $z_w = \mathbf{w}_w'^{\top} \bar{\mathbf{h}}$ is high when the two vectors point in a similar direction and have large magnitudes. Words whose output vectors point in a similar direction to $\bar{\mathbf{h}}$ receive high scores. After sufficient training, the output vector for the correct center word will align closely with the averaged context embedding.

The output matrix $\mathbf{W}'$ encodes a different kind of information than $\mathbf{W}$. Each column of $\mathbf{W}'$ is a "context signature" for that word: the direction in embedding space from which the word is most easily predicted. This is subtly different from the word's embedding in $\mathbf{W}$, which represents how the word behaves when it is a context word. As a consequence, the two matrices can diverge. In practice, researchers have found that averaging $\mathbf{W}$ and $\mathbf{W}'^{\top}$ sometimes produces better embeddings than using either alone, because the two matrices capture complementary distributional properties.

To convert raw scores into probabilities, CBOW applies the softmax function. For each vocabulary word $w$:

where:

• $P(w \mid C)$: the probability that $w$ is the center word given context $C$
• $z_w = \mathbf{w}_w'^{\top} \bar{\mathbf{h}}$: the raw score for word $w$, the dot product of $w$'s output vector and the averaged context embedding
• $\sum_{w'=1}^{V} \exp(z_{w'})$: the normalizing constant summing over all $V$ vocabulary words

The exponential function ensures all probabilities are positive, and dividing by the sum ensures they add up to one. Words with higher dot products with $\bar{\mathbf{h}}$ receive exponentially higher probability, making the distribution sharply peaked around the highest-scoring word.

The sum in the denominator is the computational bottleneck. With a vocabulary of 100,000 words, every single forward pass requires computing 100,000 dot products. This is why full softmax training is impractical for large vocabularies, and approximations like negative sampling are essential for real-world use.

For a single training example with context $C$ and true center word $w_t$, we want to maximize $P(w_t \mid C)$. Across all training examples, we maximize the log-likelihood:

Expanding the softmax, this becomes:

which can be rewritten as:

where:

• $\mathbf{w}_{w_t}'^{\top} \bar{\mathbf{h}}$: the score of the correct center word, which we want to be as large as possible
• $\log \sum_{w'=1}^{V} \exp(\mathbf{w}_{w'}'^{\top} \bar{\mathbf{h}})$: the log-sum-exp over all words, which is a normalizing constant

This decomposition is illuminating. The first term rewards increasing the score for the true center word. The second term penalizes any increase in the scores for all words (including the true one), because the log-sum-exp grows when any score grows. The result is a tug-of-war: the model must push up the score for the true word while keeping the scores for all other words low. This is exactly the competitive pressure that shapes the embedding space.

In practice, we minimize the negative log-likelihood (equivalent to minimizing cross-entropy loss), so the loss for one training example is:

With random initialization, probabilities are roughly uniform across all vocabulary words. Training will push "fox" to receive much higher probability for this context. The roughly equal initial distribution also illustrates a key feature of softmax: without meaningful parameters, every word is equally likely, which is the maximum-entropy distribution. Training breaks this symmetry by adjusting parameters so that the correct word becomes more probable.

To train CBOW with gradient descent, we need to compute how the loss changes with respect to every parameter. The parameters are the embedding matrix $\mathbf{W}$ and the output matrix $\mathbf{W}'$. Understanding these gradients is important for implementing the model and for developing intuition about what happens during training and why the averaging operation produces the regularizing effect we described earlier.

The loss for one training example is $L = -\log P(w_t \mid C)$. Expanding this through the softmax, the gradient with respect to the output score $z_w$ (before softmax) is the prediction error:

where:

• $P(w \mid C)$: the model's predicted probability for word $w$
• $\mathbf{1}[w = w_t]$: 1 if $w$ is the true center word, 0 otherwise

This is the standard softmax gradient, and it has a beautiful interpretation. The model is right when $P(w_t \mid C) \approx 1$ (error near 0), and wrong when $P(w_t \mid C) \approx 0$ (error near 1). For all non-target words, the error is $P(w \mid C) - 0 = P(w \mid C)$, meaning the model is penalized for assigning any probability to wrong words. The gradient is zero only when the model assigns probability 1 to the correct word and 0 to all others, an ideal that is never achieved in practice.

This prediction-error form has a deep connection to maximum likelihood estimation. The gradient of the negative log-likelihood with respect to the pre-softmax logits is exactly the difference between the predicted distribution and the one-hot target distribution. This clean form is one reason softmax cross-entropy is the standard objective for multiclass classification.

Since $z_w = \mathbf{w}_w'^{\top} \bar{\mathbf{h}}$, the gradient with respect to the output column vector $\mathbf{w}_w'$ is:

In matrix form, the full gradient for $\mathbf{W}'$ is the outer product of the prediction error vector and the hidden representation:

where $\mathbf{e} = \mathbf{p} - \mathbf{y}$ is the error vector ($\mathbf{p}$ is the softmax probability vector and $\mathbf{y}$ is the one-hot target vector).

The key insight here is that the update to $\mathbf{W}'$ touches every column. Every column is updated, rather than only the target word.s column. Every vocabulary word's output vector is updated at every training step: the target word's column is pushed toward $\bar{\mathbf{h}}$, and every other word's column is pushed slightly away from $\bar{\mathbf{h}}$ (because their prediction errors are positive). This global coupling is what makes the vocabulary's output representations mutually competitive: they cannot all be similar to every context average.

To propagate gradients back to the embedding matrix, we first need the gradient with respect to $\bar{\mathbf{h}}$:

where:

• $\mathbf{W}'$: the output weight matrix of shape $d \times V$
• $\mathbf{e}$: the prediction error vector of shape $V \times 1$

This is a weighted sum of all the output column vectors, weighted by the prediction errors. Words with large positive errors contribute their output vectors in the direction that increases the score for those words. Words with large negative errors (the true target word with error $P(w_t) - 1$, which is negative) pull $\bar{\mathbf{h}}$ toward their output vectors.

The gradient $\frac{\partial L}{\partial \bar{\mathbf{h}}}$ can be interpreted as "the direction in which the averaged context embedding should move to reduce the loss." If the model currently assigns high probability to the wrong word "dog" and low probability to the correct word "fox," this gradient will point toward the output vector for "fox" and away from the output vector for "dog." Moving $\bar{\mathbf{h}}$ in this direction makes "fox" more predictable from the current context.

Since $\bar{\mathbf{h}} = \frac{1}{C} \sum_{i=1}^{C} \mathbf{w}_{c_i}$, and each context word embedding $\mathbf{w}_{c_i}$ contributes equally to the average, the gradient with respect to each context word's embedding is:

The $\frac{1}{C}$ factor is the derivative of the average with respect to one component. Each context word receives the same gradient signal, scaled by $\frac{1}{C}$. This is the "bag" property in action: all context words are treated identically, and the gradient for each is the same averaged signal.

This equal sharing of gradients has a regularizing effect. If the model makes an error, the blame is distributed equally across all context words. A single context word cannot receive a large, unstable gradient update because it must share the blame with all its neighbors. This dampening makes CBOW more stable during training than a model that attributes prediction errors to individual context words.

Importantly, the only rows of $\mathbf{W}$ that receive gradient updates for a given training example are the rows corresponding to the context words present in that example. Words not in the context window receive zero gradient for this example. This sparsity is computationally advantageous: even though the vocabulary may contain 100,000 words, a given training step touches at most $2m$ rows of $\mathbf{W}$ (where $m$ is the window size), regardless of vocabulary size.

The prediction error is largest for "fox" (the target), where the model needs to increase probability the most. The gradients flow back through the output matrix and distribute equally across all context word embeddings. Notice that the target word "fox" has a negative error (the model should increase its probability), while all other words have small positive errors (the model should decrease their probabilities slightly).

With the forward pass and gradients in hand, we can train CBOW using stochastic gradient descent. Each training step processes one (context, target) pair, computes the loss, backpropagates, and updates both weight matrices.

Stochastic gradient descent updates parameters after each individual training example rather than accumulating gradients over the entire dataset. For word embedding training, this is essential: the dataset is too large to load into memory, and the stochastic updates provide a stream of small gradient signals that collectively shape the embedding space. Each update is noisy (a single training example is an imperfect estimate of the true gradient), but across millions of examples the noise averages out and the embeddings converge.

The learning rate controls the magnitude of each update. Starting with a relatively high learning rate and decaying it as training progresses is standard practice in Word2Vec. Early in training, large updates quickly move embeddings from their random initializations into meaningful regions. Later in training, smaller updates fine-tune the embeddings without overshooting.

The loss curve shows the typical pattern for Word2Vec training: rapid early improvement as the model captures the most common patterns, followed by slower refinement. On small corpora the final loss is still relatively high because the model cannot distinguish many fine-grained contexts with limited data. Training on a real corpus of millions or billions of tokens would continue pushing the loss lower, and the embeddings would become much more precisely positioned.

A practical note on initialization: CBOW parameters should be initialized with small random values, not zeros. If all embeddings start at zero, the gradient for every context word is identical (they all produce the same averaged vector), and the symmetry is never broken. Small random values ensure that different words receive different initial gradients, allowing them to diverge into distinct regions of the embedding space.

The output matrix $\mathbf{W}'$ is typically initialized with the same small-random scheme. Some implementations initialize $\mathbf{W}'$ to zeros, relying on the non-zero $\mathbf{W}$ embeddings to break symmetry through the backward pass. Both approaches work in practice, though small random initialization for both matrices tends to converge slightly faster in our experience.

One of the most important differences between CBOW and Skip-gram is not architectural but statistical: what they count as one "training example." This difference ripples through training speed, embedding quality, and practical applicability, making it one of the key decisions in the Word2Vec framework.

Skip-gram treats each (center, context) word pair as an independent training example. For a window of size 2, each center word generates up to 4 separate training pairs (2 from the left, 2 from the right). For a corpus of $N$ tokens, Skip-gram generates approximately $N \times 2m$ training pairs where $m$ is the window size.

CBOW instead creates one training example for each center word position, using all $2m$ context words simultaneously. This means CBOW generates roughly $N$ training examples from the same corpus, while Skip-gram generates approximately $N \times 2m$.

For a window size of 2, Skip-gram generates roughly 4 times as many training examples as CBOW from the same corpus. For window size 5, the ratio grows to 10. This multiplicative factor has enormous practical consequences.

This difference directly affects training speed. Skip-gram performs more gradient updates per sentence because each context position generates a separate update. CBOW performs fewer updates but each update incorporates more information (averaging $2m$ context embeddings).

In practice, CBOW trains faster than Skip-gram on the same corpus because:

1. CBOW processes fewer training examples (roughly $2m$ times fewer)
2. Each CBOW forward pass is cheap since averaging is a simple mean operation
3. The gradient update for CBOW writes to fewer parameters per step (only the context words present in the window)

For large vocabularies, this speed advantage can be substantial. On a 100-billion-word corpus, the difference between 4 hours and 20 hours of training is significant. This is why CBOW became the standard choice for applications where embeddings need to be retrained frequently or where computational resources are limited.

However, faster training does not mean better embeddings. The Skip-gram model's additional training examples provide richer gradient signal, particularly for rare words. The quality difference between the two models depends on the vocabulary distribution and the specific downstream task.

The averaging step affects how well each model represents different types of words in a subtle but important way.

Skip-gram trains each word independently as a center word, allowing the model to specialize its representation based on the word's unique distribution of contexts. A rare word that only appears 10 times still has its own dedicated training signal from those 10 examples. Each appearance generates 4 separate gradient updates (for window size 2), yielding 40 total updates for the rare word's embedding. These updates may be noisy, but they are specifically targeted at the rare word's actual co-occurrence patterns.

CBOW averages context embeddings before making a prediction. This averaging provides a regularizing effect: the signal for any one context word is diluted by the other context words. For frequent words (which appear in many different contexts and receive many gradient updates), this regularization is beneficial, smoothing out noise. For rare words (which appear infrequently and may have atypical contexts), the averaging can obscure the signal.

Consider a concrete example. Suppose "serendipity" appears 5 times in the training corpus. In CBOW, each appearance contributes one training example where "serendipity" is a context word averaging with 3 other context words. The gradient update for "serendipity" is thus diluted by a factor of 4 (for window size 2). Skip-gram, by contrast, would generate 5 training examples where "serendipity" is the center word and each surrounding word is the target, plus additional examples where "serendipity" appears as a context word. The signal for "serendipity" is stronger and less diluted.

This leads to a practical guideline:

• Use CBOW when your application needs fast training, handles primarily frequent words, or works with a large corpus where training speed matters.
• Use Skip-gram when your application requires good representations for rare or specialized vocabulary.

Let us trace through a complete CBOW training step for the example sentence, confirming that all the pieces fit together.

Consider the sentence "The quick brown fox jumps over the lazy dog" with target word "fox" and context ["quick", "brown", "jumps", "over"] (window size 2). We will perform one complete gradient descent step and verify the loss decreases. This walkthrough connects the mathematical derivations above to the concrete numerical operations that happen in every training iteration.

The procedure has four stages. First, the forward pass computes $\bar{\mathbf{h}}$ from the context embeddings and then computes the probability distribution over the vocabulary using the output matrix and softmax. Second, the loss measures how far the current distribution is from the ideal (all probability on "fox"). Third, the backward pass computes gradients for all parameters. Fourth, the update subtracts the scaled gradients from the current parameters.

After a single gradient step, the probability assigned to "fox" increases while probabilities for incorrect words decrease. The loss decreases, confirming that the gradient step moved the parameters in the right direction. This is the fundamental loop of CBOW training: observe the prediction error, compute gradients that would reduce the error, and take a small step in that direction. Repeat this across millions of training examples, and the embeddings gradually organize themselves so that words with similar distributions end up in similar regions of the space.

Notice that the improvement from a single step is modest: the probability of "fox" might increase from 0.10 to 0.12, while the loss decreases by a fraction of a nit. This is expected and desirable. Large updates would overfit to individual training examples. The embedding space is shaped by the aggregate of millions of small updates, each nudging the representations slightly toward the global structure encoded in the corpus.

After training, the embedding matrix $\mathbf{W}$ contains meaningful word representations. Even on a small corpus, words that appear in similar contexts cluster together in embedding space. We can project the 8-dimensional embeddings down to 2D using PCA to visualize these relationships.

Principal Component Analysis (PCA) finds the directions of maximum variance in the embedding space and projects the data onto them. The first principal component captures the most variance, and the second captures the next most. A 2D PCA projection preserves as much structure as possible while making the data visualizable. The resulting scatter plot does not show the full geometry (much information is lost in reducing 8 dimensions to 2), but it reveals the broad clustering structure that the model has learned.

Even with this small corpus and brief training, words with similar grammatical roles and contextual distributions tend to cluster together. The nouns "fox" and "dog" should occupy nearby positions because both appear as subjects and objects in similar sentence structures. Adjectives like "quick," "brown," and "lazy" should cluster because they all appear before nouns. With a larger corpus and more training, these clusters become much tighter and semantically meaningful. The embeddings capture the statistical regularities of the text, placing words in positions that reflect their distributional behavior.

The behavior of CBOW is controlled by several important hyperparameters. Choosing these correctly has a significant impact on the quality of the resulting embeddings.

Embedding dimension ($d$): The dimensionality of the word vectors. Common values range from 50 to 300. Higher dimensions can capture more nuance but require more training data and memory. On a small corpus, high-dimensional embeddings tend to overfit: the model has more degrees of freedom than the data can constrain, and the extra dimensions fill with noise. On large corpora (billions of words), embeddings of 300 dimensions consistently outperform smaller sizes. The sweet spot depends on corpus size and downstream task requirements.

Window size ($m$): The number of words on each side of the center to include as context. Larger windows capture broader topical context because words that appear in the same general topic area will co-occur within the wider window even if they are not directly adjacent. Smaller windows capture tighter syntactic context: adjectives and their nouns, verbs and their subjects. For tasks that depend on syntactic similarity (parsing, part-of-speech tagging), smaller windows (1 or 2) tend to work better. For tasks that depend on semantic similarity (topic modeling, information retrieval), larger windows (5 or more) tend to be superior.

Learning rate: The step size for gradient descent updates. Too large causes divergence; too small causes slow convergence. Typically initialized around 0.025 and decayed during training. The original Word2Vec implementation uses a linear decay schedule: starting at 0.025 and linearly decreasing to 0.0001 as training progresses. This adaptive schedule allows aggressive initial learning while stabilizing the final embeddings.

Vocabulary size ($V$): Determined by the corpus. Larger vocabularies require more memory for the output matrix $\mathbf{W}'$ (size $d \times V$) and make the softmax sum more expensive to compute. With $V = 100{,}000$ and $d = 200$, the output matrix requires roughly 80 MB of memory. For practical applications with full softmax, vocabulary sizes above 100,000 become computationally prohibitive, which motivates the approximations covered in the next chapter.

Minimum count: Words appearing fewer than this many times in the corpus are excluded from the vocabulary. Setting min_count to 5 is common, filtering out rare words that have insufficient data to train well. These rare words receive so few gradient updates that their embeddings remain neededly random. This provides no useful information. Excluding them reduces vocabulary size, speeds up training, and avoids including low-quality embeddings in later applications.

The hyperparameter sensitivity plot illustrates how window size and embedding dimension affect the training trajectory. Wider windows and larger embedding dimensions both reduce the final training loss on this corpus, but they also slow each training epoch because wider windows produce more context combinations and larger matrices require more computation. In production, these tradeoffs must be balanced against available compute and the requirements of the downstream task.

The original Word2Vec paper introduced an important technique called subsampling (or frequency-based downsampling) that significantly improves embedding quality. The idea is simple: very frequent words like "the," "is," and "a" appear in almost every context and provide little useful signal about the meaning of their neighbors. Instead of using every occurrence of these words in training, the model randomly discards them with a probability that increases with word frequency.

The subsampling probability for word $w$ with frequency $f(w)$ in the corpus is:

where:

• $f(w)$: the fraction of the corpus occupied by word $w$ (frequency divided by total tokens)
• $t$: a threshold hyperparameter, typically set to $10^{-5}$

Words with frequency below $t$ are never discarded. Words with frequency much greater than $t$ are discarded with high probability. The threshold $t = 10^{-5}$ means that any word making up more than 1 in 100,000 tokens is a candidate for subsampling. In a typical English corpus, this catches words like "the" (which might appear in 5% of all positions) very aggressively, while leaving most content words untouched.

Subsampling has two distinct benefits. First, it speeds up training by reducing the number of training examples from very common words that provide little discriminative signal. Second, it improves embedding quality: when "the" is discarded from a context window, the remaining context words form more semantically meaningful co-occurrences. Instead of "fox" co-occurring with "the" (a useless relationship), "fox" is more likely to co-occur with semantically rich words that happen to appear within the same window after "the" is removed.

Another practical technique from the original implementation is dynamic window sizing. Rather than always using the maximum window size $m$, the model samples the actual window size uniformly from $\{1, 2, \ldots, m\}$ for each training example.

This produces an interesting statistical effect. Words immediately adjacent to the center word are included in every window size from 1 to $m$, while words at the maximum distance are only included when the window size is sampled as $m$. Words at distance $k$ are included with probability $(m - k + 1) / m$.

The result is a soft weighting: nearby words contribute more frequently to training examples than distant words, without requiring any explicit weighting scheme. This implicitly encodes the linguistic intuition that a word's immediate neighbors carry more information about its meaning than words several positions away. Dynamic window sizing is enabled by default in the original Word2Vec implementation and is one reason the original CBOW embeddings outperformed naive implementations that used fixed window sizes.

The full softmax in CBOW sums over every vocabulary word in the denominator during training. With a vocabulary of 100,000 or more words, this is computationally expensive: every gradient step must compute $V$ dot products and update many rows of $\mathbf{W}'$. This is the primary bottleneck for training on large corpora.

To put the scale in perspective, consider training on the Google News corpus (100 billion words) with a vocabulary of 3 million words and embedding dimension 300. Each forward pass requires 3 million dot products of 300-dimensional vectors. Each backward pass requires updating 3 million column vectors of the output matrix. With billions of training examples, this full-softmax approach is computationally infeasible.

In practice, this is addressed through two approximations: negative sampling and hierarchical softmax. Negative sampling replaces the sum over all $V$ words with a sum over a small number of randomly selected "negative" words, reducing the per-step cost from $O(V)$ to $O(k)$ where $k$ is typically 5 to 20. Hierarchical softmax represents the vocabulary as a binary tree and computes probabilities as products along tree paths, also reducing per-step cost to $O(\log V)$. These approximations are covered in detail in the next chapter.

The averaging operation, while elegant, discards positional information entirely. "The dog bit the man" and "The man bit the dog" would produce identical context representations for the center word "bit" because the context words are the same, just in different positions. Similarly, "Alice knows Bob" and "Bob knows Alice" have the same bag of context words but entirely different meanings from the perspective of the subject and object.

This means CBOW cannot learn different representations for semantically distinct orderings. A word in subject position cannot be distinguished from the same word in object position. For many NLP tasks, this positional blindness is an acceptable tradeoff for the speed advantage. But for tasks that require understanding syntactic structure or argument roles, positional information is essential.

More expressive architectures overcome this limitation. LSTMs preserve word order by processing sequences left-to-right and right-to-left, building up a state that depends on the entire preceding context. Transformers use positional encodings to give the model explicit access to word positions, and attention mechanisms allow the model to weigh different context words differently based on both their content and position. These models produce contextual embeddings where the same word receives different representations in different sentence positions, a capability CBOW fundamentally lacks.

CBOW, like all Word2Vec variants, produces static embeddings: each word type (distinct word form) receives exactly one vector in the embedding matrix. This is problematic for polysemous words, words with multiple distinct meanings.

The word "bank" can refer to a financial institution, the side of a river, or the act of tilting an aircraft. In CBOW's embedding space, all these senses are collapsed into a single vector that represents some kind of average over all the contexts in which "bank" appears. This averaged vector may not represent any single meaning well. In downstream tasks like word sense disambiguation or named entity recognition, the conflated embedding provides mixed signals.

This limitation motivated the development of contextualized word representations, where the embedding of a word depends on the sentence context in which it appears. ELMo (Embeddings from Language Models) was an early approach that ran a language model over the full sentence and used hidden states as contextualized embeddings. BERT and GPT later demonstrated that transformer-based language models produce rich contextual representations that capture polysemy and context-sensitivity in ways that static embeddings cannot. We will cover these models in depth in later chapters.

Despite these limitations, CBOW's contribution to NLP cannot be overstated. The Word2Vec paper demonstrated that simple neural models could produce word embeddings that captured analogical relationships (king - man + woman = queen) in a way that no prior method had achieved. This went beyond a qualitative demonstration: the embeddings quantifiably outperformed prior approaches on a suite of analogy and similarity tasks.

CBOW's speed made it practical to train on the web-scale corpora needed to learn these relationships. Earlier neural word embedding methods (like the language models of Bengio et al. from 2003) were correct in principle but too slow to scale to the data volumes required for high-quality embeddings. CBOW's linear architecture and the approximations developed for it (negative sampling, subsampling) made billion-word training feasible.

The embeddings produced by Word2Vec became standard features for almost every NLP task from 2013 to 2018. Sentiment analysis, named entity recognition, machine translation, question answering, and text classification all benefited from pre-trained Word2Vec embeddings as starting points. Researchers no longer needed to train embeddings from scratch for every task: the pre-trained vectors captured general linguistic knowledge that could be fine-tuned for specific applications. This transfer learning paradigm, initiated in earnest by Word2Vec, became the dominant approach in NLP and eventually gave rise to the massive pre-trained language models that define the field today.

GloVe, FastText, and dependency-based embeddings all refined the ideas introduced by CBOW and Skip-gram. GloVe reframed the objective as matrix factorization of co-occurrence counts. This provides a stronger theoretical foundation. FastText extended the model to use subword information, capturing morphological structure that pure word-level models miss. Dependency-based embeddings used syntactic dependency paths rather than linear windows, producing embeddings more sensitive to grammatical structure. Each of these methods acknowledged its debt to Word2Vec while addressing one of its specific weaknesses.

The CBOW model learns word embeddings by training a neural network to predict a center word from its surrounding context. The key concepts are:

• Context averaging: CBOW averages the embedding vectors of all context words into a single representation $\bar{\mathbf{h}}$ before making a prediction. This is what makes CBOW a "bag of words" model, trading positional information for computational efficiency and stability.
• Objective function: The model maximizes the log-probability of the correct center word given the context, equivalent to minimizing the negative log-likelihood of the training data. The full softmax over all $V$ vocabulary words is expensive and motivates the approximations in the next chapter.
• Gradient derivation: Gradients flow from the output layer back through the averaging operation, distributing equally to all context word embeddings with a factor of $\frac{1}{C}$. This shared gradient creates the regularizing effect that benefits frequent words.
• Training speed: CBOW generates fewer training examples per word than Skip-gram (one per center position versus $2m$ per center position), making it faster to train on large corpora but providing weaker signal for rare words.
• Frequent vs. rare words: CBOW's averaging helps frequent words by smoothing noisy contexts, but can disadvantage rare words by diluting their limited training signal. Skip-gram is preferred when rare vocabulary quality matters.
• Practical techniques: Subsampling frequent words and using dynamic window sizes significantly improve both training speed and embedding quality beyond the basic algorithm described here.
• Static embeddings: CBOW produces one vector per word type, which cannot capture polysemy or context-sensitivity. Contextual embedding methods like BERT overcome this fundamental limitation at much greater computational cost.

The next chapter covers negative sampling, an approximation technique that eliminates the expensive full softmax computation and enables Word2Vec to train on truly large-scale corpora.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the CBOW model and Word2Vec word embeddings.