Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Negative Sampling › The Sampling Distribution: Why

Michael BrenndoerferApril 4, 20256 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 21Negative Sampling › The Sampling Distribution: Why

Reading section 5 of 15 · The Sampling Distribution: Why · 6 minSection 5 of 15 · 6 min

Section 5 of 15 · 6 min read

Negative Sampling › The Sampling Distribution: WhyOne section · Sections in this chapter5 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The choice of how to sample negative words turns out to be surprisingly consequential. A naive approach might sample words uniformly at random. But consider what happens: "the" appears in nearly every sentence, yet with uniform sampling it would be drawn as a negative in only $1/V$ of cases. Meanwhile, "aardvark" appears rarely in real text but would be sampled just as often as "the."

This mismatch creates problems. Common function words occur near almost every word in the corpus, so sampling them as negatives is confusing because they are often legitimate context words too. Rare words sampled too often as negatives cause the model to push everything away from them, hurting their embedding quality.

The Word2Vec authors discovered that a modified unigram distribution works best:

P_n(w) = \frac{f(w)^{0.75}}{\sum_{w'} f(w')^{0.75}}

where:

$P_n(w)$ : probability of sampling word $w$ as a negative sample
$f(w)$ : frequency (count) of word $w$ in the corpus
$f(w)^{0.75}$ : the frequency raised to the power 0.75, which compresses the range
$\sum_{w'} f(w')^{0.75}$ : normalizing constant summing over all vocabulary words $w'$
$0.75$ : smoothing exponent determined empirically by the Word2Vec authors

The key insight is that raising to a power less than 1 compresses the range of values. If word A appears 10,000 times and word B appears 10 times, their raw frequency ratio is 1000:1. After applying the 0.75 exponent, $10000^{0.75} / 10^{0.75} \approx 178$ . The ratio shrinks from 1000:1 to about 178:1, boosting the representation of rare words without ignoring frequency entirely.

In[13]:

Code

def create_sampling_distribution(word_counts, alpha=0.75):
    """
    Create the negative sampling distribution using smoothed unigram frequencies.

    Args:
        word_counts: Dictionary mapping words to their corpus frequencies
        alpha: Exponent to smooth the distribution (default 0.75)

    Returns:
        words: List of words
        probs: Sampling probabilities
    """
    words = list(word_counts.keys())
    freqs = np.array([word_counts[w] for w in words], dtype=np.float64)
    smoothed = freqs**alpha
    probs = smoothed / smoothed.sum()
    return words, probs


# Example corpus frequencies
word_frequencies = {
    "the": 100000,
    "a": 80000,
    "is": 50000,
    "word": 1000,
    "embedding": 500,
    "neural": 400,
    "semantic": 200,
    "vector": 800,
    "aardvark": 5,
    "quixotic": 2,
}

words, probs = create_sampling_distribution(word_frequencies, alpha=0.75)

Out[14]:

Console

Comparison of Sampling Distributions:
-----------------------------------------------------------------
Word           Raw Freq    Uniform    Freq Prob     Smoothed
-----------------------------------------------------------------
the             100,000     0.1000     0.429356     0.393092
a                80,000     0.1000     0.343485     0.332516
is               50,000     0.1000     0.214678     0.233734
word              1,000     0.1000     0.004294     0.012431
embedding           500     0.1000     0.002147     0.007391
neural              400     0.1000     0.001717     0.006252
semantic            200     0.1000     0.000859     0.003718
vector              800     0.1000     0.003435     0.010515
aardvark              5     0.1000     0.000021     0.000234
quixotic              2     0.1000     0.000009     0.000118

Sum of smoothed probs: 1.0000

The comparison shows how smoothing redistributes probability mass. Common words like "the" drop from dominating the distribution (raw frequency ~43%) to a more moderate share. Rare words like "aardvark" and "quixotic" see their sampling probability increase relative to their raw frequency. This ensures they receive adequate training signal.

Out[15]:

Visualization

Bar chart on log scale showing raw frequency probabilities dominated by high-frequency words. — Raw frequency sampling distribution. Common words like 'the' account for over 40% of samples, dominating training signal.

Bar chart on log scale showing smoothed probabilities with reduced gap between frequent and rare words. — Raw frequency sampling distribution. Common words like 'the' account for over 40% of samples, dominating training signal.

Bar chart showing ratio of smoothed to raw probability, with green bars for rare words and red bars for frequent words. — Raw frequency sampling distribution. Common words like 'the' account for over 40% of samples, dominating training signal.

Why 0.75? Intuition and AlternativesLink Copied

The exponent 0.75 was found empirically by the Word2Vec authors. Consider the intuition by looking at the extremes. An exponent of 1.0 (raw frequency) means frequent words dominate: "the" might account for 7% of samples, wasting computation on a word the model has already learned to handle. An exponent of 0.0 (uniform) treats rare words as often as frequent ones, but rare words share contexts with fewer other words, making them less informative negatives. The value 0.75 strikes a balance: common words still appear often enough to learn good embeddings, while rare words get meaningful representation in the negative samples.

Out[16]:

Visualization

Line plot on log scale showing sampling probabilities for each word under five different exponents from 0.0 to 1.0. — Effect of the smoothing exponent on sampling probability across words sorted by frequency. As the exponent decreases from 1.0 (raw frequency, orange) toward 0.0 (uniform, purple), the distribution flattens. The chosen exponent 0.75 (green) reduces the gap between frequent and rare words while still respecting the overall frequency ordering. The log scale reveals the full range of probabilities across words spanning five orders of magnitude in raw frequency.