CBOW Model › Subsampling Frequent Words

The original Word2Vec paper introduced an important technique called subsampling (or frequency-based downsampling) that significantly improves embedding quality. The idea is simple: very frequent words like "the," "is," and "a" appear in almost every context and provide little useful signal about the meaning of their neighbors. Instead of using every occurrence of these words in training, the model randomly discards them with a probability that increases with word frequency.

The subsampling probability for word $w$ with frequency $f(w)$ in the corpus is:

where:

• $f(w)$: the fraction of the corpus occupied by word $w$ (frequency divided by total tokens)
• $t$: a threshold hyperparameter, typically set to $10^{-5}$

Words with frequency below $t$ are never discarded. Words with frequency much greater than $t$ are discarded with high probability. The threshold $t = 10^{-5}$ means that any word making up more than 1 in 100,000 tokens is a candidate for subsampling. In a typical English corpus, this catches words like "the" (which might appear in 5% of all positions) very aggressively, while leaving most content words untouched.

Subsampling has two distinct benefits. First, it speeds up training by reducing the number of training examples from very common words that provide little discriminative signal. Second, it improves embedding quality: when "the" is discarded from a context window, the remaining context words form more semantically meaningful co-occurrences. Instead of "fox" co-occurring with "the" (a useless relationship), "fox" is more likely to co-occur with semantically rich words that happen to appear within the same window after "the" is removed.