CBOW Model › Architecture: Context Averaging

The two weight matrices in CBOW mirror those in Skip-gram:

• Embedding matrix $\mathbf{W}$ (size $V \times d$): Each row is the $d$-dimensional embedding for one vocabulary word. When CBOW receives context words, it looks up each context word's row and averages them.
• Output matrix $\mathbf{W}'$ (size $d \times V$): Each column corresponds to a vocabulary word's "context representation." This matrix converts the averaged embedding into a score vector over the full vocabulary.

These two matrices serve distinct roles during training even though both are learned simultaneously. The embedding matrix $\mathbf{W}$ is the primary product: these are the word vectors you use after training. The output matrix $\mathbf{W}'$ is more of a scaffolding: it provides the contrastive signal needed to push word embeddings apart during training. Some practitioners use $\mathbf{W}'$ as an alternative embedding matrix (averaging the two is sometimes beneficial), but in most applications only $\mathbf{W}$ is retained.

You might wonder why CBOW has no activation function between the embedding lookup and the output layer. Skip-gram also omits nonlinearities. This is a deliberate choice, and it carries practical effects.

Nonlinearities like ReLU or sigmoid introduce computational cost and complicate backpropagation. More importantly, they change the geometry of the embedding space. A linear architecture means the learned embeddings live in a space where vector arithmetic is meaningful. The famous Word2Vec property that king - man + woman = queen holds precisely because the embedding space is linear. Adding a nonlinearity would destroy this arithmetic structure.

There is also a theoretical argument from information theory. The task of predicting a word from its context is a self-supervised objective that forces the model to compress distributional information into a fixed-size vector. The compression happens through the bottleneck of the embedding dimension $d$, not through a nonlinearity. As long as $d$ is much smaller than $V$, the model must find a compact representation of the word's distributional properties, and that representation turns out to be geometrically meaningful.