GloVe › Using Pretrained GloVe Vectors
Language AI HandbookAll chapters GloVe › Using Pretrained GloVe VectorsOne section · Sections in this chapter10 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
In practice, you rarely train GloVe from scratch. Pretrained vectors from the Stanford NLP Group, trained on billions of tokens from Wikipedia and Common Crawl, are far more useful for most applications. The pretrained vectors have seen vocabulary and co-occurrence statistics that no small-corpus training run can match.
# Install gensim if needed
# !uv pip install gensim --quiet
import gensim.downloader as api
# Load pretrained GloVe vectors (50d, trained on 6B tokens from Wikipedia+Gigaword)
glove_model = api.load("glove-wiki-gigaword-50")Vocabulary size: 400,000 Vector dimension: 50 Sample words: ['the', ',', '.', 'of', 'to', 'and', 'in', 'a', '"', "'s"]
# Find most similar words
similar_to_king = glove_model.most_similar("king", topn=8)
similar_to_science = glove_model.most_similar("science", topn=8)Most similar to 'king': prince 0.824 queen 0.784 ii 0.775 emperor 0.774 son 0.767 uncle 0.763 kingdom 0.754 throne 0.754 Most similar to 'science': sciences 0.855 research 0.844 institute 0.839 studies 0.837 physics 0.831 psychology 0.829 scientific 0.829 biology 0.828
# Test the classic analogy: king - man + woman = ?
analogy_result = glove_model.most_similar(
positive=["king", "woman"], negative=["man"], topn=5
)king - man + woman = ? queen 0.852 throne 0.766 prince 0.759 daughter 0.747 elizabeth 0.746
The analogy task works by vector arithmetic in the embedding space. "king - man + woman" computes the vector for "king", subtracts the vector offset between male and female (encoded as ), and finds the nearest neighbor in vocabulary. The fact that "queen" emerges near the top of this list demonstrates that GloVe's objective successfully encodes semantic relationships as directional offsets in the embedding geometry.
GloVe Captures Co-Occurrence Ratios
Let's verify the original motivating insight: that GloVe embeddings encode co-occurrence ratios as geometric relationships.
# Compute cosine similarities as a proxy for the ratio P(k|ice)/P(k|steam)
# In embedding space, similar words have high dot product
probe_words = ["solid", "gas", "water", "fashion", "cold", "hot"]
target_pairs = [("ice", "steam")]
for t1, t2 in target_pairs:
similarities = {}
for probe in probe_words:
if probe in glove_model:
sim_t1 = glove_model.similarity(t1, probe)
sim_t2 = glove_model.similarity(t2, probe)
similarities[probe] = (sim_t1, sim_t2, sim_t1 - sim_t2)Probe sim(ice,k) sim(steam,k) difference -------------------------------------------------- solid 0.523 0.409 0.115 gas 0.592 0.590 0.001 water 0.684 0.639 0.045 fashion 0.299 0.060 0.239 cold 0.642 0.490 0.152 hot 0.742 0.554 0.188
Words like "solid" and "cold" show much higher cosine similarity to "ice" than to "steam" in the embedding space, while "water" is roughly equidistant. This is exactly the pattern predicted by the co-occurrence ratios in the paper's motivating table. The geometry of the learned space directly reflects the statistical structure of the corpus.
Key Parameters for Pretrained and Custom Training
When using GloVe, either loading pretrained vectors or training from scratch, the key parameters to understand are:
- embed_dim: Embedding dimensionality. Values of 50 to 300 work well for most tasks. Larger dimensions capture more nuance but require more training data and memory. The Stanford pretrained vectors come in 50d, 100d, 200d, and 300d variants.
- window: Context window size for building the co-occurrence matrix. Larger windows (5 to 10) capture broad topical and semantic relationships. Smaller windows (1 to 3) capture syntactic patterns more tightly.
- x_max: The co-occurrence count at which the weighting function saturates to 1. Pairs with counts above this threshold receive full weight. The paper recommends 100 for large corpora, but smaller values suit smaller corpora.
- alpha: The exponent in the weighting function. The paper recommends 0.75, which was empirically validated across multiple analogy and similarity benchmarks.
- learning_rate: AdaGrad learning rate. The paper uses 0.05. AdaGrad's adaptive nature makes this less sensitive than in standard SGD.
- n_epochs: Number of training passes over all nonzero co-occurrence pairs. Typically 50 to 100 epochs is sufficient for convergence on large corpora.