Word2Vec Training › The Negative Sampling Distribution

Before moving to learning rate scheduling, it is worth examining a key implementation detail from the previous chapter that has direct consequences for training quality: the distribution from which negative samples are drawn.

Recall from the Negative Sampling chapter that the training objective uses $k$ randomly selected "negative" words as contrast for each positive (center, context) pair. The key question is: what distribution should we use to sample those negatives?

Sampling uniformly from the vocabulary is a natural first guess. Every word has an equal chance of being selected as a negative. But this leads to poor training dynamics. Common words already dominate the training corpus through their positive co-occurrence statistics. If we also sample them frequently as negatives, we update their embeddings constantly, which makes them dominant attractors in the gradient surface. Rare words, which appear infrequently as positives, also appear infrequently as negatives under uniform sampling, so they receive very little gradient signal in total.

Mikolov et al.'s solution is to sample from a smoothed unigram distribution, specifically the unigram distribution raised to the power $3/4$:

where $f(w)$ is the raw count (or frequency) of word $w$.

The $3/4$ exponent is a carefully chosen compromise. Raising to a power less than 1 compresses the frequency distribution: it reduces the probability of very common words and increases the probability of rare words relative to uniform sampling. The value $3/4$ was empirically found to work well by the original authors, and it has been confirmed across many subsequent studies. To see the effect, consider a vocabulary where "the" has frequency 50,000 and "photosynthesis" has frequency 10. Under raw frequency sampling, "the" would be selected 5,000 times more often than "photosynthesis". Under unigram$^{3/4}$ sampling, the ratio shrinks to $50000^{0.75} / 10^{0.75} \approx 1778 / 5.6 \approx 317$ times. Rare words are still sampled less often, but not to the extreme degree that their raw frequency difference would suggest.

This smoothing is important because it determines how evenly gradient updates are distributed across the vocabulary. Every time a word appears as a negative sample, its context embedding receives a gradient update pushing it to produce lower dot products with center word embeddings. If rare words almost never appear as negatives, their context embeddings barely move from initialization. The $3/4$ exponent ensures that even rare words accumulate meaningful gradient signal as negatives over the course of training.