Skip-gram Model › Softmax Over the Vocabulary

To train the model, we need to convert the raw scores $u_j$ into a probability distribution over all vocabulary words. The softmax function accomplishes this by exponentiating all scores and normalizing:

$$P(w_O | w_I) = \frac{\exp(u_{w_O})}{\sum_{j=1}^{V} \exp(u_j)}$$

where:

• $P(w_O | w_I)$: the probability of observing context word $w_O$ given center word $w_I$
• $u_{w_O} = \mathbf{v}'_{w_O} \cdot \mathbf{h}$: the dot-product score for context word $w_O$
• $\sum_{j=1}^{V} \exp(u_j)$: the normalizing constant (the partition function), summing exponential scores over all $V$ vocabulary words

The exponential function ensures all values are positive. Dividing by the sum ensures the values sum to 1, making the output a valid probability distribution. Words with higher dot-product scores with the center word receive higher probability mass. This is the mechanism by which training works: for a true (center, context) pair, we want the probability of the true context word to be high. We want the model to assign most probability mass to the word observed in context, which means the embedding for the true context word should have a high dot product with the center word embedding.

The softmax has a useful geometric interpretation: it measures alignment between the center word's embedding and each candidate context embedding. A context word whose embedding points in a similar direction to the center word embedding receives a high dot product and therefore high probability. As training progresses, the context embedding $\mathbf{v}'_{w_O}$ for a word that frequently appears near $w_I$ gets pushed to be more similar to $\mathbf{v}_{w_I}$. Words that never appear together receive gradients pushing their embeddings apart.

The denominator $\sum_{j=1}^{V} \exp(u_j)$ requires computing a score for every word in the vocabulary. For typical vocabularies of 100,000 to 1,000,000 words, this is extremely expensive. For each training example, you compute $V$ dot products between the center word embedding and every context embedding in $\mathbf{W}'$. For a corpus with billions of training pairs, this sum dominates the computation. A vocabulary of 1,000,000 words means 1,000,000 dot products per training step, totaling trillions of floating point operations across a full training run.

This bottleneck is not a quirk of a particular implementation choice; it is inherent to any exact softmax over a large vocabulary. The reason is that softmax is a global operation: you must query every word in order to normalize correctly. This is in contrast to classification problems with a small number of classes, where softmax is cheap.

This fundamental computational bottleneck is why the Negative Sampling and Hierarchical Softmax approximations were introduced alongside Skip-gram in the original Word2Vec paper. The theoretical objective described in this chapter is the correct formulation; in practice, you always use one of these approximations. The next chapters cover each approximation in detail, showing how they preserve the essential learning dynamics while making training tractable on realistic corpora.