Word2Vec Training › Comparing Gensim and PyTorch Results

Both implementations are training the same underlying model, but they differ in how they handle subsampling, the exact negative sampling implementation, and initialization. Comparing the similarity structures they learn reveals how sensitive embeddings are to implementation details.

This sensitivity is an important practical insight. Two runs of Word2Vec on the same corpus with different random seeds will produce embeddings that are rotations of each other in embedding space (word similarities will match, but the actual vector coordinates will differ). Two runs with different hyperparameters or implementation details will produce embeddings with different similarity structures. There is no single "ground truth" set of word embeddings: the geometry depends on the training choices.

This is not a bug but a feature. The training objective defines what we want to be similar (words that appear in similar contexts), but it does not define a unique solution. Many configurations of the embedding space satisfy the training objective equally well. SGD with random initialization finds one of these configurations, and different initializations or slightly different training procedures find different ones. For most downstream applications, any well-trained configuration works equally well because the relative structure (which words are near each other) is preserved even if the absolute coordinates differ.