Negative Sampling › Limitations and Impact

Negative sampling made Word2Vec practical, but it comes with limitations worth understanding.

The most fundamental limitation is that negative sampling does not optimize the true softmax objective. It optimizes a surrogate that happens to produce good embeddings for most downstream tasks, but tasks requiring precise probability estimates or calibrated outputs may be better served by exact methods. The theoretical connection to NCE provides some justification for why the surrogate works, but the guarantees are weaker.

The hyperparameters $k$ and $\alpha$ require tuning. The defaults from the original paper (k=5 to 10, $\alpha$=0.75) work well across many settings, but they are not universally optimal. Low-resource languages, specialized technical vocabularies, or domains with very skewed frequency distributions may benefit from different choices.

Negative sampling also inherits the fundamental limitation of all Word2Vec models: it produces static embeddings. Each word receives a single fixed vector regardless of context. "Bank" in a financial document and "bank" near a river share the same embedding, even though they mean different things. This limitation motivated the development of contextual embedding models like ELMo, BERT, and GPT, which represent the next generation of word representations. We will explore these models in later parts of the book.

Despite these limitations, negative sampling's impact on NLP research was enormous. It made embedding training practical at billion-word scale, enabling the empirical discoveries about embedding geometry (analogies, semantic arithmetic) that captured the community's imagination. It influenced the design of subsequent models, and the core idea of binary discrimination objectives reappears in modern contrastive learning methods that train vision models, cross-modal systems, and sentence encoders. The same principle that makes negative sampling work for word embeddings powers frameworks like SimCLR, CLIP, and many others: rather than computing expensive normalizations over large output spaces, learn by distinguishing real examples from carefully sampled negatives.