GloVe › GloVe: Word Embeddings via Co-Occurrence Matrix Factorization

Word2Vec and its variants (Skip-gram, CBOW) learn embeddings by training a neural network to predict words from their context. The embeddings emerge as a byproduct of that prediction task: the network never directly sees the corpus statistics it is using; it only sees individual context windows one at a time. But there is an older, more direct tradition in NLP for capturing word relationships: count how often words co-occur in the same context across a large corpus, then factorize the resulting co-occurrence matrix into dense vectors. This approach has deep roots in Latent Semantic Analysis from the 1990s, but it fell out of favor when prediction-based methods demonstrated cleaner results on analogy benchmarks.

GloVe (Global Vectors for Word Representation), introduced by Pennington, Socher, and Manning at Stanford in 2014, bridges these two traditions. It brings the global, corpus-wide statistics of matrix factorization methods together with the efficient, scalable training machinery of prediction-based methods like Word2Vec. The result is a model that is theoretically cleaner than Word2Vec, often faster to train, and produces word vectors of comparable or better quality on standard benchmarks.

The key insight behind GloVe is deceptively simple: word co-occurrence ratios carry more signal than raw co-occurrence counts. If "ice" and "steam" are both related to "water", their co-occurrence probabilities with "water" will be similar. But if you look at how each co-occurs with "solid", "ice" wins by a large margin. The ratio $P(\text{solid} | \text{ice}) / P(\text{solid} | \text{steam})$ reveals something meaningful: "solid" is much more associated with ice than with steam. GloVe's objective function is designed to directly capture these ratios in the geometry of the embedding space.

This chapter derives GloVe's objective function from first principles, shows every algebraic step without shortcuts, explains the weighted least squares formulation and its design choices, connects GloVe to classical matrix factorization and PMI, covers the bias terms and weighting function, and compares GloVe to Word2Vec in terms of training dynamics, performance, and practical use. We then implement GloVe from scratch and explore pretrained vectors.