Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

CBOW Model › Context Word Averaging

Michael BrenndoerferApril 3, 20257 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 20CBOW Model › Context Word Averaging

Reading section 4 of 15 · Context Word Averaging · 7 minSection 4 of 15 · 7 min

Section 4 of 15 · 7 min read

CBOW Model › Context Word AveragingOne section · Sections in this chapter4 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The averaging step is CBOW's defining operation. How should the model combine information from multiple context words into a single vector? CBOW takes the simplest possible answer: element-wise mean. If the context words "quick," "brown," "jumps," and "over" each contribute a $d$ -dimensional embedding, the averaged representation is the vector whose $j$ -th element is the mean of all four embeddings' $j$ -th elements.

More formally, given $C$ context words with indices $c_1, c_2, \ldots, c_C$ in the vocabulary, the hidden representation is:

\bar{\mathbf{h}} = \frac{1}{C} \sum_{i=1}^{C} \mathbf{w}_{c_i}

where:

$\bar{\mathbf{h}}$ : the averaged hidden representation, a $d$ -dimensional vector representing the "centroid" of the context in embedding space
$C$ : the number of context words (typically $2m$ for a window of size $m$ , but fewer at sentence boundaries)
$\mathbf{w}_{c_i}$ : the embedding of the $i$ -th context word, the row at index $c_i$ in the embedding matrix $\mathbf{W}$

This averaging has an important geometric interpretation. Each embedding vector corresponds to a point in $d$ -dimensional space. The average corresponds to the centroid of those points. If the context words are semantically coherent (which they tend to be in natural language), their centroid points toward the region of the embedding space where the center word should live.

To see why this works, consider what the model must learn for the centroid to be useful. Each context word embedding must encode the word's own meaning and information about what words typically appear near it. After training, the centroid of "quick," "brown," "jumps," and "over" should point toward the region where nouns like "fox" live, because that is the region the model was repeatedly trained to predict from such context centroids.

There is an alternative design choice worth understanding: instead of averaging, you could sum the context embeddings. The sum is mathematically equivalent to the average (they differ only by a constant scaling factor), but the average has a practical advantage: the magnitude of the hidden representation $\bar{\mathbf{h}}$ does not grow with the number of context words. This stability is helpful for maintaining consistent gradient magnitudes during training regardless of window size or sentence boundary effects.

In[8]:

Code

import numpy as np

# Small vocabulary for demonstration
small_vocab = ["the", "quick", "brown", "fox", "jumps", "over", "lazy", "dog"]
word_to_idx = {word: i for i, word in enumerate(small_vocab)}
idx_to_word = {i: word for word, i in word_to_idx.items()}

# Initialize small embedding matrix (random, untrained)
embedding_dim = 4
np.random.seed(42)
W = np.random.randn(len(small_vocab), embedding_dim) * 0.5


def average_context_embeddings(context_words, W, word_to_idx):
    """Average the embedding vectors of a set of context words."""
    embeddings = np.array([W[word_to_idx[word]] for word in context_words])
    return np.mean(embeddings, axis=0)


context = ["quick", "brown", "jumps", "over"]
avg_embedding = average_context_embeddings(context, W, word_to_idx)

Out[9]:

Console

Context Word Averaging (embedding_dim=4):
----------------------------------------------------
Context words: ['quick', 'brown', 'jumps', 'over']

Individual embeddings:
  quick   : [-0.117, -0.117, +0.790, +0.384]
  brown   : [-0.235, +0.271, -0.232, -0.233]
  jumps   : [-0.506, +0.157, -0.454, -0.706]
  over    : [+0.733, -0.113, +0.034, -0.712]

Averaged embedding (h-bar):
  h_bar   : [-0.031, +0.050, +0.034, -0.317]

Verification (manual column averages):
  Dim 1: mean(['-0.117', '-0.235', '-0.506', '+0.733']) = -0.031
  Dim 2: mean(['-0.117', '+0.271', '+0.157', '-0.113']) = +0.050
  Dim 3: mean(['+0.790', '-0.232', '-0.454', '+0.034']) = +0.034
  Dim 4: mean(['+0.384', '-0.233', '-0.706', '-0.712']) = -0.317

The averaged embedding is not the embedding of any particular word. It is a synthetic point in the embedding space that represents the combined semantic context. After training, this averaged vector will be positioned such that the center word's row in $\mathbf{W}'$ receives the highest score.

Out[10]:

Visualization

Heatmap of 4 context word embeddings and their average, showing values across 4 embedding dimensions. — Heatmap of context word embeddings and their element-wise average for the context of 'fox'. Each row shows a context word's 4-dimensional embedding, and the bottom row shows the averaged representation. The thick horizontal line separates individual embeddings from the computed average. The averaged values lie between the extremes of the individual embeddings, capturing a blended centroid of the context that CBOW uses to predict the target word.

The heatmap makes the averaging operation concrete. Each column of the upper four rows contains the embedding values for one dimension. The bottom row shows how the average smooths these values, placing the centroid between the extremes. In this random-initialization example, the values carry no semantic meaning yet, but the same averaging process applies once the embeddings have been trained. After training, the average of "quick," "brown," "jumps," and "over" will be a vector that the output matrix can reliably decode as "fox."

The Geometric View of AveragingLink Copied

It is worth dwelling on the geometry a little longer because it explains both the strengths and the limitations of CBOW. In $d$ -dimensional embedding space, each word occupies a point. The average of $C$ points is the centroid, the center of mass of the configuration. When the $C$ context words form a tight cluster (because they are semantically similar or syntactically related), the centroid is a stable, representative point. When the context words are semantically diverse or scattered across the embedding space, the centroid can end up in an unusual region, away from any natural cluster.

This geometry explains CBOW.s stability on frequent words. Frequent words appear in many different contexts. Over many training examples, the model encounters "fox" predicted from contexts like ["quick", "brown", "jumps", "over"], ["cunning", "red", "ran", "through"], ["the", "chased", "by", "hound"], and many more. Each of these averages lands near the "fox" region of the space, and the gradients reinforce the embedding accordingly. The signal is consistent and strong.

Rare words, by contrast, may appear in only a handful of contexts. If "dhole" (a type of wild canid) appears in contexts mostly involving predation and forests, its averaged context vectors may cluster in a region that also corresponds to many other animals. The signal is weaker, and neighboring words may not be reliable discriminators. This is why Skip-gram, which trains on each context word individually, tends to outperform CBOW on rare vocabulary.