Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

CBOW Model › The CBOW Objective Function

Michael BrenndoerferApril 3, 20255 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 20CBOW Model › The CBOW Objective Function

Reading section 5 of 15 · The CBOW Objective Function · 5 minSection 5 of 15 · 5 min

Section 5 of 15 · 5 min read

CBOW Model › The CBOW Objective FunctionOne section · Sections in this chapter5 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

With the averaged context representation in hand, CBOW needs to assign high probability to the correct center word and low probability to all other vocabulary words. This is a multiclass classification problem across the entire vocabulary.

From Hidden Representation to ScoresLink Copied

After computing $\bar{\mathbf{h}}$ , the model computes a score for every vocabulary word by multiplying $\bar{\mathbf{h}}$ against the output matrix $\mathbf{W}'$ :

\mathbf{z} = \mathbf{W}'^{\top} \bar{\mathbf{h}}

where:

$\mathbf{z}$ : a vector of raw scores (logits), one per vocabulary word, shape $V \times 1$
$\mathbf{W}'$ : the output weight matrix of shape $d \times V$
$\bar{\mathbf{h}}$ : the averaged context embedding of shape $d \times 1$

Each element $z_w$ of this score vector measures how well the averaged context aligns with word $w$ 's output representation. The dot product $z_w = \mathbf{w}_w'^{\top} \bar{\mathbf{h}}$ is high when the two vectors point in a similar direction and have large magnitudes. Words whose output vectors point in a similar direction to $\bar{\mathbf{h}}$ receive high scores. After sufficient training, the output vector for the correct center word will align closely with the averaged context embedding.

The output matrix $\mathbf{W}'$ encodes a different kind of information than $\mathbf{W}$ . Each column of $\mathbf{W}'$ is a "context signature" for that word: the direction in embedding space from which the word is most easily predicted. This is subtly different from the word's embedding in $\mathbf{W}$ , which represents how the word behaves when it is a context word. As a consequence, the two matrices can diverge. In practice, researchers have found that averaging $\mathbf{W}$ and $\mathbf{W}'^{\top}$ sometimes produces better embeddings than using either alone, because the two matrices capture complementary distributional properties.

The Softmax DistributionLink Copied

To convert raw scores into probabilities, CBOW applies the softmax function. For each vocabulary word $w$ :

P(w \mid C) = \frac{\exp(z_w)}{\sum_{w'=1}^{V} \exp(z_{w'})}

where:

$P(w \mid C)$ : the probability that $w$ is the center word given context $C$
$z_w = \mathbf{w}_w'^{\top} \bar{\mathbf{h}}$ : the raw score for word $w$ , the dot product of $w$ 's output vector and the averaged context embedding
$\sum_{w'=1}^{V} \exp(z_{w'})$ : the normalizing constant summing over all $V$ vocabulary words

The exponential function ensures all probabilities are positive, and dividing by the sum ensures they add up to one. Words with higher dot products with $\bar{\mathbf{h}}$ receive exponentially higher probability, making the distribution sharply peaked around the highest-scoring word.

The sum in the denominator is the computational bottleneck. With a vocabulary of 100,000 words, every single forward pass requires computing 100,000 dot products. This is why full softmax training is impractical for large vocabularies, and approximations like negative sampling are essential for real-world use.

Training Objective: Maximize Log-LikelihoodLink Copied

For a single training example with context $C$ and true center word $w_t$ , we want to maximize $P(w_t \mid C)$ . Across all training examples, we maximize the log-likelihood:

J = \sum_{(C, w_t)} \log P(w_t \mid C)

Expanding the softmax, this becomes:

\log P(w_t \mid C) = \log \frac{\exp(\mathbf{w}_{w_t}'^{\top} \bar{\mathbf{h}})}{\sum_{w'=1}^{V} \exp(\mathbf{w}_{w'}'^{\top} \bar{\mathbf{h}})}

which can be rewritten as:

\log P(w_t \mid C) = \mathbf{w}_{w_t}'^{\top} \bar{\mathbf{h}} - \log \sum_{w'=1}^{V} \exp(\mathbf{w}_{w'}'^{\top} \bar{\mathbf{h}})

where:

$\mathbf{w}_{w_t}'^{\top} \bar{\mathbf{h}}$ : the score of the correct center word, which we want to be as large as possible
$\log \sum_{w'=1}^{V} \exp(\mathbf{w}_{w'}'^{\top} \bar{\mathbf{h}})$ : the log-sum-exp over all words, which is a normalizing constant

This decomposition is illuminating. The first term rewards increasing the score for the true center word. The second term penalizes any increase in the scores for all words (including the true one), because the log-sum-exp grows when any score grows. The result is a tug-of-war: the model must push up the score for the true word while keeping the scores for all other words low. This is exactly the competitive pressure that shapes the embedding space.

In practice, we minimize the negative log-likelihood (equivalent to minimizing cross-entropy loss), so the loss for one training example is:

L = -\log P(w_t \mid C)

In[11]:

Code

def softmax(x):
    """Numerically stable softmax."""
    x = x - np.max(x)  # Subtract max for numerical stability
    e_x = np.exp(x)
    return e_x / e_x.sum()


def cbow_forward(context_words, W, W_prime, word_to_idx):
    """
    CBOW forward pass.
    Returns averaged hidden representation and softmax probabilities.
    """
    # Step 1: Look up embeddings for context words
    context_embs = np.array([W[word_to_idx[w]] for w in context_words])

    # Step 2: Average context embeddings
    h_bar = np.mean(context_embs, axis=0)

    # Step 3: Compute output scores
    scores = W_prime.T @ h_bar

    # Step 4: Apply softmax
    probs = softmax(scores)

    return h_bar, probs


# Initialize model parameters
V = len(small_vocab)
d = 4
np.random.seed(42)
W = np.random.randn(V, d) * 0.1
W_prime = np.random.randn(d, V) * 0.1

# Run forward pass
h_bar, probs = cbow_forward(context, W, W_prime, word_to_idx)

Out[12]:

Console

CBOW Forward Pass:
--------------------------------------------------
Context: ['quick', 'brown', 'jumps', 'over']
Averaged h_bar: [-0.0063, 0.0099, 0.0069, -0.0634]

Output probabilities (before training):
  dog     : 0.1258 ########################
  lazy    : 0.1257 ########################
  the     : 0.1255 ########################
  over    : 0.1250 ########################
  quick   : 0.1249 ########################
  jumps   : 0.1248 ########################
  brown   : 0.1244 ########################
  fox     : 0.1239 ########################

P('fox' | context) = 0.1239
(Random init, so probabilities should be roughly uniform)

With random initialization, probabilities are roughly uniform across all vocabulary words. Training will push "fox" to receive much higher probability for this context. The roughly equal initial distribution also illustrates a key feature of softmax: without meaningful parameters, every word is equally likely, which is the maximum-entropy distribution. Training breaks this symmetry by adjusting parameters so that the correct word becomes more probable.