CBOW Model › CBOW Gradient Derivation

To train CBOW with gradient descent, we need to compute how the loss changes with respect to every parameter. The parameters are the embedding matrix $\mathbf{W}$ and the output matrix $\mathbf{W}'$. Understanding these gradients is important for implementing the model and for developing intuition about what happens during training and why the averaging operation produces the regularizing effect we described earlier.

The loss for one training example is $L = -\log P(w_t \mid C)$. Expanding this through the softmax, the gradient with respect to the output score $z_w$ (before softmax) is the prediction error:

$$\frac{\partial L}{\partial z_w} = P(w \mid C) - \mathbf{1}[w = w_t]$$

where:

• $P(w \mid C)$: the model's predicted probability for word $w$
• $\mathbf{1}[w = w_t]$: 1 if $w$ is the true center word, 0 otherwise

This is the standard softmax gradient, and it has a beautiful interpretation. The model is right when $P(w_t \mid C) \approx 1$ (error near 0), and wrong when $P(w_t \mid C) \approx 0$ (error near 1). For all non-target words, the error is $P(w \mid C) - 0 = P(w \mid C)$, meaning the model is penalized for assigning any probability to wrong words. The gradient is zero only when the model assigns probability 1 to the correct word and 0 to all others, an ideal that is never achieved in practice.

This prediction-error form has a deep connection to maximum likelihood estimation. The gradient of the negative log-likelihood with respect to the pre-softmax logits is exactly the difference between the predicted distribution and the one-hot target distribution. This clean form is one reason softmax cross-entropy is the standard objective for multiclass classification.

Since $z_w = \mathbf{w}_w'^{\top} \bar{\mathbf{h}}$, the gradient with respect to the output column vector $\mathbf{w}_w'$ is:

$$\frac{\partial L}{\partial \mathbf{w}_w'} = \left(P(w \mid C) - \mathbf{1}[w = w_t]\right) \bar{\mathbf{h}}$$

In matrix form, the full gradient for $\mathbf{W}'$ is the outer product of the prediction error vector and the hidden representation:

$$\frac{\partial L}{\partial \mathbf{W}'} = \bar{\mathbf{h}} \cdot \mathbf{e}^{\top}$$

where $\mathbf{e} = \mathbf{p} - \mathbf{y}$ is the error vector ($\mathbf{p}$ is the softmax probability vector and $\mathbf{y}$ is the one-hot target vector).

The key insight here is that the update to $\mathbf{W}'$ touches every column. Every column is updated, rather than only the target word.s column. Every vocabulary word's output vector is updated at every training step: the target word's column is pushed toward $\bar{\mathbf{h}}$, and every other word's column is pushed slightly away from $\bar{\mathbf{h}}$ (because their prediction errors are positive). This global coupling is what makes the vocabulary's output representations mutually competitive: they cannot all be similar to every context average.

To propagate gradients back to the embedding matrix, we first need the gradient with respect to $\bar{\mathbf{h}}$:

$$\frac{\partial L}{\partial \bar{\mathbf{h}}} = \mathbf{W}' \mathbf{e}$$

where:

• $\mathbf{W}'$: the output weight matrix of shape $d \times V$
• $\mathbf{e}$: the prediction error vector of shape $V \times 1$

This is a weighted sum of all the output column vectors, weighted by the prediction errors. Words with large positive errors contribute their output vectors in the direction that increases the score for those words. Words with large negative errors (the true target word with error $P(w_t) - 1$, which is negative) pull $\bar{\mathbf{h}}$ toward their output vectors.

The gradient $\frac{\partial L}{\partial \bar{\mathbf{h}}}$ can be interpreted as "the direction in which the averaged context embedding should move to reduce the loss." If the model currently assigns high probability to the wrong word "dog" and low probability to the correct word "fox," this gradient will point toward the output vector for "fox" and away from the output vector for "dog." Moving $\bar{\mathbf{h}}$ in this direction makes "fox" more predictable from the current context.

Since $\bar{\mathbf{h}} = \frac{1}{C} \sum_{i=1}^{C} \mathbf{w}_{c_i}$, and each context word embedding $\mathbf{w}_{c_i}$ contributes equally to the average, the gradient with respect to each context word's embedding is:

$$\frac{\partial L}{\partial \mathbf{w}_{c_i}} = \frac{1}{C} \cdot \frac{\partial L}{\partial \bar{\mathbf{h}}}$$

The $\frac{1}{C}$ factor is the derivative of the average with respect to one component. Each context word receives the same gradient signal, scaled by $\frac{1}{C}$. This is the "bag" property in action: all context words are treated identically, and the gradient for each is the same averaged signal.

This equal sharing of gradients has a regularizing effect. If the model makes an error, the blame is distributed equally across all context words. A single context word cannot receive a large, unstable gradient update because it must share the blame with all its neighbors. This dampening makes CBOW more stable during training than a model that attributes prediction errors to individual context words.

Importantly, the only rows of $\mathbf{W}$ that receive gradient updates for a given training example are the rows corresponding to the context words present in that example. Words not in the context window receive zero gradient for this example. This sparsity is computationally advantageous: even though the vocabulary may contain 100,000 words, a given training step touches at most $2m$ rows of $\mathbf{W}$ (where $m$ is the window size), regardless of vocabulary size.