GloVe › Code Implementation
Language AI HandbookAll chapters GloVe › Code ImplementationOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement GloVe from scratch: build a co-occurrence matrix, define the weighted least squares objective, and train word vectors with PyTorch. We will use a small toy corpus to make the computation tractable, but every component scales directly to large vocabularies.
Building the Co-Occurrence Matrix
We start by tokenizing a corpus and computing the symmetric co-occurrence matrix within a fixed window. A common refinement is inverse-distance weighting: context words farther from the center contribute a fractional count () rather than a full count. This gives more weight to immediate neighbors and less weight to distant context, capturing syntactic proximity more naturally.
from collections import Counter
# Small toy corpus for illustration
# Two semantic domains: animals/actions and states-of-water
corpus = [
"the cat sat on the mat",
"the cat ate the rat",
"the dog sat on the mat",
"the dog chased the cat",
"ice is solid and cold",
"steam is gas and hot",
"water is liquid and wet",
"ice and steam are both water",
"solid ice melts into water",
"hot steam comes from boiling water",
"the cat drinks cold water",
"the dog sat near the cold ice",
]
# Tokenize and build vocabulary
tokens_list = [sentence.lower().split() for sentence in corpus]
all_tokens = [t for sent in tokens_list for t in sent]
word_counts = Counter(all_tokens)
# Filter to words appearing at least twice
min_count = 2
vocab = sorted([w for w, c in word_counts.items() if c >= min_count])
word_to_idx = {w: i for i, w in enumerate(vocab)}
V = len(vocab)Vocabulary size: 14 Words: ['and', 'cat', 'cold', 'dog', 'hot', 'ice', 'is', 'mat', 'on', 'sat', 'solid', 'steam', 'the', 'water']
def build_cooccurrence_matrix(tokens_list, word_to_idx, window=2):
"""Build symmetric co-occurrence matrix with inverse-distance weighting."""
V = len(word_to_idx)
X = np.zeros((V, V), dtype=np.float32)
for tokens in tokens_list:
# Only keep tokens in vocabulary
indices = [word_to_idx[t] for t in tokens if t in word_to_idx]
for pos, center in enumerate(indices):
# Window: positions within +-window
start = max(0, pos - window)
end = min(len(indices), pos + window + 1)
for ctx_pos in range(start, end):
if ctx_pos == pos:
continue
ctx = indices[ctx_pos]
# Weight by inverse distance (common variant)
dist = abs(ctx_pos - pos)
X[center, ctx] += 1.0 / dist
return X
X = build_cooccurrence_matrix(tokens_list, word_to_idx, window=3)Co-occurrence matrix shape: (14, 14) Non-zero entries: 89 Sparsity: 54.6%
The high sparsity is expected: most pairs of words in the vocabulary never appear in the same context window. Real-world vocabularies of 400,000 words would be even sparser, with far less than 1% of entries being nonzero.
Visualizing the Co-Occurrence Matrix
Implementing the GloVe Model
The model has four learnable components per word: a word vector (when the word is the target), a context vector (when the word is in context), and bias scalars for each role.
class GloVeModel(nn.Module):
def __init__(self, vocab_size, embed_dim):
super().__init__()
# Word vectors (target embeddings)
self.word_embeddings = nn.Embedding(vocab_size, embed_dim)
# Context vectors (context embeddings)
self.context_embeddings = nn.Embedding(vocab_size, embed_dim)
# Bias terms for each word
self.word_biases = nn.Embedding(vocab_size, 1)
self.context_biases = nn.Embedding(vocab_size, 1)
# Initialize with small random values
nn.init.uniform_(
self.word_embeddings.weight, -0.5 / embed_dim, 0.5 / embed_dim
)
nn.init.uniform_(
self.context_embeddings.weight, -0.5 / embed_dim, 0.5 / embed_dim
)
nn.init.zeros_(self.word_biases.weight)
nn.init.zeros_(self.context_biases.weight)
def forward(self, word_ids, context_ids):
# Dot product + biases
w = self.word_embeddings(word_ids) # (batch, dim)
c = self.context_embeddings(context_ids) # (batch, dim)
bw = self.word_biases(word_ids).squeeze(1) # (batch,)
bc = self.context_biases(context_ids).squeeze(1) # (batch,)
return (w * c).sum(dim=1) + bw + bc # (batch,)The initialization of word and context vectors with small random values scaled by follows the original GloVe paper. Initializing biases to zero is natural because the bias should start as a neutral correction that the optimizer adjusts as needed.
Defining the Weighted Loss
def glove_weight_fn(x, x_max=10.0, alpha=0.75):
"""Weighting function for GloVe loss (adapted for small toy corpus)."""
return torch.clamp((x / x_max) ** alpha, max=1.0)
def glove_loss(predictions, log_counts, weights):
"""Weighted least squares loss."""
residuals = (predictions - log_counts) ** 2
return (weights * residuals).sum()Note that x_max=10.0 is used here (rather than the paper's 100) to account for the tiny corpus. With only 12 sentences, co-occurrence counts rarely exceed 10, so a lower saturation threshold is appropriate.
Preparing Training Data
# Extract nonzero co-occurrence pairs
nonzero = np.argwhere(X > 0)
word_ids = torch.tensor(nonzero[:, 0], dtype=torch.long)
context_ids = torch.tensor(nonzero[:, 1], dtype=torch.long)
counts = torch.tensor(X[nonzero[:, 0], nonzero[:, 1]], dtype=torch.float32)
log_counts = torch.log(counts)
weights = glove_weight_fn(counts, x_max=10.0)Training pairs: 89 Sample counts: [1.0, 1.0, 1.3333333730697632, 2.5, 1.0] Sample log-counts: [0.0, 0.0, 0.28768211603164673, 0.9162907600402832, 0.0] Sample weights: [0.17782793939113617, 0.17782793939113617, 0.2206500768661499, 0.3535533845424652, 0.17782793939113617]
Each training pair is an index pair, the log co-occurrence count , and the weight . The entire co-occurrence matrix is loaded into memory as a flat list of nonzero pairs, which is the standard efficient representation for sparse training data.
Training the Model
torch.manual_seed(42)
embed_dim = 8
model = GloVeModel(V, embed_dim)
optimizer = torch.optim.Adagrad(model.parameters(), lr=0.05)
losses = []
n_epochs = 200
for epoch in range(n_epochs):
optimizer.zero_grad()
preds = model(word_ids, context_ids)
loss = glove_loss(preds, log_counts, weights)
loss.backward()
optimizer.step()
losses.append(loss.item())Initial loss: 13.2993 Final loss (epoch 200): 0.0001 Reduction: 220540.5x
The log-scale plot reveals a common pattern in GloVe training: a steep initial drop followed by a much slower refinement phase. The initial drop corresponds to the vectors moving from random initialization to a rough approximation of the log co-occurrence structure. The slow refinement phase is the model fine-tuning the dot products to match the data more precisely, with AdaGrad's decreasing learning rates naturally slowing this phase down.
Extracting and Inspecting Embeddings
After training, we combine word and context vectors as the final representation. This is the averaging trick described earlier: summing the two complementary views of each word.
# Final embeddings: sum of word and context vectors
W = model.word_embeddings.weight.detach().numpy()
C = model.context_embeddings.weight.detach().numpy()
embeddings = W + C # Common practice: sum (equivalent to averaging up to scale)Cosine similarities after GloVe training: ice - cold : 0.205 cat - dog : 0.512 sat - mat : -0.662 ice - dog : -0.645
The cosine similarities reflect the semantic structure in the corpus. Words from the same semantic domain ("ice" and "cold", "cat" and "dog") should show higher similarity than cross-domain pairs ("ice" and "dog"). With only 12 sentences and 8 dimensions, the signal is weak but directionally correct.
Visualizing the Learned Embeddings
With only 8 dimensions and a tiny corpus, the embeddings will not be perfect. But projecting to 2D with PCA lets us check whether semantically similar words cluster together.
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
emb_2d = pca.fit_transform(embeddings)