GloVe › Summary

GloVe derives word embeddings by fitting vectors to reproduce the log co-occurrence statistics of a large corpus. The derivation follows from a single insight about ratios, and each step is logically necessary given the design requirements.

The key ideas are:

• Co-occurrence ratios carry more discriminative signal than raw probabilities, and GloVe's objective is built to encode these ratios in the geometry of the embedding space
• The weighted least squares objective $J = \sum_{i,j} f(X_{ij})(\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j - \log X_{ij})^2$ directly fits vectors to log co-occurrence counts, with each term weighted by co-occurrence reliability
• The functional form $F = \exp$ is forced by the requirement that $F(a - b) = F(a)/F(b)$, connecting vector differences to probability ratios
• The weighting function $f(x) = \min(1, (x/x_{\max})^\alpha)$ downweights rare and zero co-occurrences, solving both the $\log 0$ problem and noise sensitivity
• Bias terms $b_i$ and $\tilde{b}_j$ absorb marginal word-frequency effects, letting the dot product capture pure relational structure
• GloVe implicitly factorizes a shifted PMI matrix, unifying count-based and prediction-based embedding methods under a single theoretical framework
• AdaGrad adapts per-parameter learning rates, handling the extreme frequency imbalance in natural language vocabularies
• Combining word and context vectors ($\vec{w}_i + \tilde{\vec{w}}_i$) as the final representation reduces variance and slightly improves quality
• Pretrained GloVe vectors trained on billions of tokens provide strong features for downstream NLP tasks without any task-specific embedding training

The next chapter on FastText extends the word embedding idea in a different direction: instead of learning one vector per word type, it decomposes words into character n-grams, enabling representations for out-of-vocabulary words and better handling of morphologically rich languages.