Word2Vec Training › Summary

The Word2Vec training pipeline combines several ideas that work together to produce high-quality embeddings:

• Preprocessing establishes a clean vocabulary by filtering rare words with a minimum frequency threshold, lowercasing for token consolidation, and removing punctuation. These decisions directly affect how much signal each vocabulary word receives during training.

• Subsampling discards frequent words at training time using a keep probability that decreases with word frequency. This improves both speed and embedding quality by reducing the dominance of high-frequency function words and effectively giving rare content words more gradient signal per corpus token.

• Negative sampling distribution uses unigram frequency raised to the $3/4$ power, which compresses the frequency distribution to give rare words more exposure as negative samples without completely ignoring frequency differences.

• Learning rate scheduling starts high to allow rapid exploration from random initialization and decays linearly to allow fine-grained convergence. The linear schedule works across a wide range of corpus sizes and vocabulary configurations.

• Minibatch vs. online training is a tradeoff between training speed (favoring larger batches for GPU utilization) and gradient noise (which online training provides for regularization). Modern implementations use small minibatches of 256-512 pairs.

• Gensim provides a battle-tested implementation that handles all these details automatically, with near-C performance from Cython compilation and efficient streaming support for large corpora. It is the right choice for practical applications.

• PyTorch from scratch gives complete visibility into the training process, making it easier to experiment with architectural modifications, add custom loss terms, or debug unexpected behavior. The implementation directly mirrors the mathematical formulation from the Negative Sampling chapter.

The gap between knowing Word2Vec's theory and successfully training it comes down to these pipeline decisions. The next chapter turns from training mechanics to what the trained embeddings can do, examining how vector arithmetic over embeddings captures semantic relationships through word analogies.