GloVe › Training GloVe Efficiently

GloVe minimizes the weighted least squares objective using AdaGrad, an adaptive gradient descent optimizer well suited to sparse, high-dimensional problems. The update rule for word vector $\vec{w}_i$ at step $t$ for a sampled pair $(i, j)$ proceeds as follows.

Step 1: Compute the residual (prediction error):

$$r_{ij} = \vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j - \log X_{ij}$$

This is the signed error between the model's prediction and the target log co-occurrence count. A positive residual means the model predicts the pair is more similar than the data supports; a negative residual means the data shows more co-occurrence than the model currently encodes.

Step 2: Compute the gradient with respect to $\vec{w}_i$:

$$\nabla_{\vec{w}_i} \mathcal{L}_{ij} = 2 f(X_{ij}) \cdot r_{ij} \cdot \tilde{\vec{w}}_j$$

The weighting $f(X_{ij})$ scales the gradient: pairs with low weight contribute a smaller gradient, updating the vectors less aggressively for noisy co-occurrences. This links between the weighting function and the optimization: rare pairs not only contribute less to the final objective value, they also push the vectors less during each training step.

Step 3: Accumulate squared gradients (AdaGrad's adaptive memory):

$$G_i \leftarrow G_i + \left(\nabla_{\vec{w}_i} \mathcal{L}_{ij}\right)^2$$

AdaGrad maintains a separate accumulated squared gradient for each parameter dimension. Dimensions that receive large gradients accumulate large $G_i$ values.

Step 4: Update the word vector with an adaptive learning rate:

$$\vec{w}_i \leftarrow \vec{w}_i - \frac{\eta}{\sqrt{G_i + \epsilon}} \nabla_{\vec{w}_i} \mathcal{L}_{ij}$$

where:

• $\eta$: the global learning rate (typically 0.05)
• $G_i$: accumulated sum of squared gradients for word $i$ (element-wise)
• $\epsilon$: small constant for numerical stability (e.g., $10^{-8}$)

The $1/\sqrt{G_i}$ factor is AdaGrad's key feature. Dimensions that receive large gradients accumulate larger $G_i$, resulting in smaller effective learning rates. This stabilizes training when some word dimensions are updated far more frequently than others, which happens naturally in word embedding problems: high-frequency words update their vectors often while rare words update infrequently. The same updates apply symmetrically to $\tilde{\vec{w}}_j$, $b_i$, and $\tilde{b}_j$.

GloVe samples co-occurrence pairs $(i, j)$ in proportion to their weight $f(X_{ij})$, so high-weight pairs are seen more often during each epoch. Training typically runs for 50-100 epochs over all nonzero $(i, j)$ pairs.

AdaGrad was specifically designed for sparse learning problems, which is exactly the structure of GloVe training. Most words in a large vocabulary are rare: the top 1,000 words by frequency account for the majority of tokens, while the remaining 999,000 words appear only occasionally. In a single training epoch, common words like "the" and "of" will be involved in thousands of co-occurrence pairs, while rare technical terms may appear in only a handful.

Standard SGD with a fixed learning rate struggles with this imbalance: the learning rate must be small enough not to overshoot for common words, but this makes rare word updates almost imperceptibly small. AdaGrad solves this by maintaining per-parameter learning rates: rare word parameters accumulate small gradients and therefore keep large effective learning rates, while common word parameters accumulate large gradients and get smaller effective rates. The result is a more uniform convergence across the vocabulary.

The main drawback of AdaGrad is that the accumulated squared gradient $G_i$ only ever increases, so the effective learning rate only ever decreases. For long training runs, this can slow convergence to a crawl. More modern optimizers like Adam use an exponential moving average of squared gradients instead, which decays the memory of old gradients and allows the learning rate to recover. In practice, GloVe training with AdaGrad works well for the typical 50-100 epoch range before this becomes a significant issue.

The key hyperparameters for GloVe training interact in ways that are worth understanding.

Window size: The original paper uses a window of $W = 10$ (5 words on each side) for large corpora. Smaller windows (1-3) emphasize syntactic and local collocational relationships. Larger windows (10-20) emphasize topical and semantic relationships. In practice, a window of 5-10 is a reliable default for most applications. For tasks like POS tagging or parsing where local syntax matters, a narrower window often helps. For tasks like document similarity or topic modeling where broad semantics matter, a wider window is better.

Embedding dimensionality: Dimensions of 50-300 work well for most tasks. The original paper evaluated 50d, 100d, 200d, and 300d vectors and found that performance plateaus or slightly improves beyond 100d for most tasks, with diminishing returns above 300d. The choice involves a practical tradeoff: higher dimensions provide more expressive power but increase memory usage and can make downstream model training more expensive. For applications where the embedding is a fixed feature (not fine-tuned), 100d is often a good default.

Vocabulary size and minimum count: The co-occurrence matrix size scales as $O(V^2)$ in the worst case, but in practice most entries are zero. Discarding words that appear fewer than 5-10 times in the corpus reduces $V$ substantially while retaining nearly all useful semantic information. Words that appear fewer than 5 times simply do not have enough co-occurrence data to learn reliable vectors.

Corpus size: GloVe benefits substantially from large corpora. The original paper trained on Common Crawl (840 billion tokens) and Wikipedia (6 billion tokens). For smaller corpora (<100 million tokens), Word2Vec's online learning may generalize better because it can extract more signal from each individual context window.