Negative Sampling › How Many Negative Samples? The Hyperparameter
Language AI HandbookAll chapters Negative Sampling › How Many Negative Samples? The HyperparameterOne section · Sections in this chapter6 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The number of negative samples is a central hyperparameter that controls the trade-off between training signal quality and computational cost. More negatives provide stronger gradient information per positive pair but increase the work per training step linearly.
Typical values for range from 5 to 20. Smaller values (5 to 10) work well for large datasets where each word pair is seen many times. Larger values (15 to 25) help with smaller datasets or less frequent words. The original Word2Vec paper recommends 5 to 10 for large corpora and 15 to 25 for smaller ones.
The intuition behind these recommendations is straightforward. With a large corpus, each positive word pair appears many times throughout training. This provides ample signal to position that pair correctly. A modest number of negatives is sufficient to push away distractors. With a smaller corpus, each positive pair appears less frequently, so more negatives per step compensate by providing stronger repulsive signal that helps separate unrelated words even with limited data.
def estimate_training_cost(vocab_size, num_negatives, num_training_pairs):
"""
Compare computational cost of full softmax vs negative sampling.
Cost is measured in dot product operations per training step.
"""
softmax_cost = num_training_pairs * vocab_size
neg_sampling_cost = num_training_pairs * (1 + num_negatives)
speedup = softmax_cost / neg_sampling_cost
return softmax_cost, neg_sampling_cost, speedup
vocab_size = 100000
num_negatives = 10
num_training_pairs = 1_000_000_000 # 1 billion pairs
softmax_cost, neg_samp_cost, speedup = estimate_training_cost(
vocab_size, num_negatives, num_training_pairs
)Computational Cost Comparison: ------------------------------------------------------- Vocabulary size: 100,000 Training pairs: 1,000,000,000 Negative samples k: 10 Full softmax cost: 100,000,000,000,000 dot products Neg sampling cost: 11,000,000,000 dot products Speedup factor: 9,091x This is why negative sampling enables practical training!
For a 100,000-word vocabulary with 10 negative samples, negative sampling performs approximately 9,000 times fewer operations per training step than full softmax. What would take months with softmax can be completed in days with negative sampling.
The trade-off between the number of negative samples and computational cost is linear: doubling doubles the work per training step. But the relationship between and embedding quality is sublinear, with diminishing returns as increases. The sweet spot at to captures most of the quality gains while keeping training times manageable.