Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Negative Sampling › Practical Considerations

Michael BrenndoerferApril 4, 20253 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 21Negative Sampling › Practical Considerations

Reading section 12 of 15 · Practical Considerations · 3 minSection 12 of 15 · 3 min

Section 12 of 15 · 3 min read

Negative Sampling › Practical ConsiderationsOne section · Sections in this chapter12 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Subsampling Frequent WordsLink Copied

Very frequent words like "the" and "a" provide limited information because they appear in nearly every context. They don't help distinguish different semantic meanings. Word2Vec subsamples frequent words by randomly discarding some occurrences during training based on their frequency:

P(\text{keep } w) = \sqrt{\frac{t}{f(w)}} + \frac{t}{f(w)}

where:

$P(\text{keep } w)$ : probability of keeping an occurrence of word $w$ during training
$f(w)$ : relative frequency of word $w$ in the corpus, computed as count $(w)$ / total word count
$t$ : subsampling threshold, typically $10^{-5}$
$\sqrt{t/f(w)}$ : dominant term for very frequent words, which provides aggressive subsampling
$t/f(w)$ : additive term that prevents the probability from dropping too sharply

The formula has an intuitive interpretation: when $f(w) \gg t$ (very frequent words), $P(\text{keep } w) \approx \sqrt{t/f(w)}$ , which is much less than 1. If "the" has relative frequency $f = 0.05$ and $t = 10^{-5}$ , then $P(\text{keep}) \approx \sqrt{10^{-5}/0.05} \approx 0.014$ , meaning only about 1.4% of "the" occurrences are kept. When $f(w) \leq t$ (rare words), the formula yields values above 1, which are clamped to 1. This ensures rare words always contribute.

In[36]:

Code

def subsample_prob(word_freq, total_words, threshold=1e-5):
    """
    Compute probability of keeping a word during training.

    Frequent words are subsampled to reduce their dominance.
    """
    freq_ratio = word_freq / total_words
    if freq_ratio > threshold:
        keep_prob = np.sqrt(threshold / freq_ratio) + (threshold / freq_ratio)
        return min(keep_prob, 1.0)
    return 1.0


total = sum(word_frequencies.values())
subsample_probs = {
    w: subsample_prob(f, total) for w, f in word_frequencies.items()
}

Out[37]:

Console

Subsampling Probabilities:
--------------------------------------------------
Word          Frequency    Keep Prob
--------------------------------------------------
the             100,000       0.0048
a                80,000       0.0054
is               50,000       0.0069
word              1,000       0.0506
vector              800       0.0569
embedding           500       0.0729
neural              400       0.0821
semantic            200       0.1196
aardvark              5       1.0000
quixotic              2       1.0000

The most frequent words like "the" and "a" have very low keep probabilities, meaning they are discarded during most training iterations. Rarer words like "aardvark" and "quixotic" have keep probability 1.0. This ensures every occurrence contributes to their embeddings.

Key ParametersLink Copied

The key parameters for negative sampling training are:

num_negatives (k): Number of negative samples per positive pair. Typical range is 5 to 10 for large datasets and 15 to 25 for smaller ones. More negatives improve quality with diminishing returns and increase training time linearly.
alpha: Smoothing exponent for the noise distribution. The recommended value is 0.75. Values closer to 1.0 give more weight to frequent words; values closer to 0 approach uniform sampling.
learning_rate: Initial learning rate. The Word2Vec implementation uses 0.025 by default with linear decay to 0.0001 over training.
window_size: Context window size. Larger windows capture broader topical similarity; smaller windows capture tighter syntactic relationships.