Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Word2Vec Training › Subsampling Frequent Words

Michael BrenndoerferApril 7, 20254 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 23Word2Vec Training › Subsampling Frequent Words

Reading section 3 of 14 · Subsampling Frequent Words · 4 minSection 3 of 14 · 4 min

Section 3 of 14 · 4 min read

Word2Vec Training › Subsampling Frequent WordsOne section · Sections in this chapter3 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of the most impactful training tricks in Word2Vec is subsampling of frequent words, introduced in Mikolov et al.'s 2013 paper. The intuition is simple: words like "the", "a", and "is" appear so frequently that they provide little useful signal for learning word meaning. When "the" appears 1,000 times near "fox", those 1,000 co-occurrences tell you almost nothing specific about "fox", because "the" appears near everything.

The subsampling procedure randomly discards occurrences of frequent words during training. The decision to keep or discard each token is made independently and randomly. Each word $w$ is kept with probability:

P(\text{keep} \mid w) = \min\!\left(1, \sqrt{\frac{t}{f(w)}} + \frac{t}{f(w)}\right)

where:

$f(w)$ : the frequency of word $w$ in the corpus, expressed as a fraction of total tokens
$t$ : a threshold hyperparameter, typically $10^{-5}$ to $10^{-3}$

To understand this formula, consider the two extreme cases. When $f(w) \ll t$ , meaning the word is very rare relative to the threshold, both terms inside the $\min$ become large and the probability caps at 1: rare words are always kept. When $f(w) \gg t$ , meaning the word is very common, the probability approaches $\sqrt{t/f(w)}$ , which decreases as frequency increases. For "the" in English with frequency around 0.05 and $t = 10^{-5}$ , the keep probability is approximately $\sqrt{10^{-5}/0.05} \approx 0.014$ : "the" is kept only about 1.4% of the time.

The $\sqrt{\cdot}$ in the formula is deliberate. A pure $t/f(w)$ formula would discard very common words so aggressively that they almost vanish from training entirely. The square root gives a gentler curve: extremely common words are still discarded most of the time, but not quite so thoroughly that they contribute zero signal. In practice, even a heavily subsampled "the" still provides some co-occurrence information, and the square root formula preserves that at the margin.

In[7]:

Code

def compute_subsample_probs(vocab, total_tokens, t=1e-4):
    """
    Compute the probability of KEEPING each word during subsampling.

    Words with frequency far below t are always kept (prob = 1.0).
    Very frequent words are kept rarely (prob close to 0).
    """
    probs = {}
    for word, count in vocab.items():
        freq = count / total_tokens
        # Mikolov's formula: P(keep) = min(1, sqrt(t/f) + t/f)
        keep_prob = min(1.0, (np.sqrt(freq / t) + 1) * (t / freq))
        probs[word] = keep_prob
    return probs


total_tokens = len(tokens)
keep_probs = compute_subsample_probs(vocab, total_tokens, t=1e-4)

Out[8]:

Console

Subsampling probabilities (probability of KEEPING a word occurrence):
------------------------------------------------------------
  'the            ' freq=0.3333  keep_prob=0.0176
  'fox            ' freq=0.1429  keep_prob=0.0272
  'dog            ' freq=0.1429  keep_prob=0.0272
  'quick          ' freq=0.0476  keep_prob=0.0479
  'brown          ' freq=0.0476  keep_prob=0.0479
  'lazy           ' freq=0.0476  keep_prob=0.0479
  'was            ' freq=0.0476  keep_prob=0.0479
  'clever         ' freq=0.0476  keep_prob=0.0479
  'word           ' freq=0.0476  keep_prob=0.0479
  'embeddings     ' freq=0.0476  keep_prob=0.0479
  'in             ' freq=0.0476  keep_prob=0.0479

In[9]:

Code

def subsample_tokens(tokens, keep_probs):
    """Randomly discard frequent words according to keep_probs."""
    return [t for t in tokens if np.random.random() < keep_probs[t]]


# Apply subsampling
np.random.seed(42)
tokens_subsampled = subsample_tokens(tokens, keep_probs)

Out[10]:

Console

Original token count: 42
After subsampling:    1
Retention rate:       2.38%

Subsampling has two benefits beyond just filtering noise. First, it reduces training time because the total number of training pairs shrinks. On web-scale corpora, removing the most frequent ~1,000 words (which together account for roughly 80% of all tokens in English) can reduce training time by 50-70% without hurting embedding quality for content words.

Second, and more subtly, subsampling balances the gradient updates that rare words receive. Without subsampling, "the" would appear in roughly 50,000 training pairs per million corpus tokens. "photosynthesis" might appear in only 10. Every gradient update involving "the" either directly updates its embedding or, through negative sampling, competes against it as a negative sample. The model spends enormous capacity on function words that carry no meaning. With subsampling, "the" is reduced to perhaps 700 appearances per million tokens, much closer in effective frequency to meaningful content words. Rare words effectively receive more weight in the learning process relative to common words, which is exactly what we want.

There is a third benefit that is less often discussed: subsampling effectively widens the context window for remaining words. If we have a sequence "the quick brown fox" and we discard "the", the window around "quick" now reaches "brown" and "fox" without "the" cluttering the middle. The remaining words in the context see a wider effective neighborhood of content words, strengthening the semantic signal.