CBOW Model › Summary

The CBOW model learns word embeddings by training a neural network to predict a center word from its surrounding context. The key concepts are:

• Context averaging: CBOW averages the embedding vectors of all context words into a single representation $\bar{\mathbf{h}}$ before making a prediction. This is what makes CBOW a "bag of words" model, trading positional information for computational efficiency and stability.
• Objective function: The model maximizes the log-probability of the correct center word given the context, equivalent to minimizing the negative log-likelihood of the training data. The full softmax over all $V$ vocabulary words is expensive and motivates the approximations in the next chapter.
• Gradient derivation: Gradients flow from the output layer back through the averaging operation, distributing equally to all context word embeddings with a factor of $\frac{1}{C}$. This shared gradient creates the regularizing effect that benefits frequent words.
• Training speed: CBOW generates fewer training examples per word than Skip-gram (one per center position versus $2m$ per center position), making it faster to train on large corpora but providing weaker signal for rare words.
• Frequent vs. rare words: CBOW's averaging helps frequent words by smoothing noisy contexts, but can disadvantage rare words by diluting their limited training signal. Skip-gram is preferred when rare vocabulary quality matters.
• Practical techniques: Subsampling frequent words and using dynamic window sizes significantly improve both training speed and embedding quality beyond the basic algorithm described here.
• Static embeddings: CBOW produces one vector per word type, which cannot capture polysemy or context-sensitivity. Contextual embedding methods like BERT overcome this fundamental limitation at much greater computational cost.

The next chapter covers negative sampling, an approximation technique that eliminates the expensive full softmax computation and enables Word2Vec to train on truly large-scale corpora.