Word Embeddings: Word2Vec, GloVe and Vector Semantics › Code Implementation: Training Word Embeddings from Scratch
Word Embeddings: Word2Vec, GloVe and Vector Semantics › Code Implementation: Training Word Embeddings from ScratchOne section · Sections in this article6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Now let's implement Word2Vec skip-gram with negative sampling from scratch. This implementation will help you understand exactly how embeddings are learned.
Implementing Skip-gram
import numpy as np
class Word2Vec:
def __init__(
self,
vocab_size,
embedding_dim=100,
window_size=5,
negative_samples=5,
learning_rate=0.01,
):
self.vocab_size = vocab_size
self.embedding_dim = embedding_dim
self.window_size = window_size
self.negative_samples = negative_samples
self.learning_rate = learning_rate
# Initialize embedding matrices
# W_in: input embeddings (what we'll use as word embeddings)
# W_out: output embeddings (for predicting context words)
self.W_in = np.random.randn(vocab_size, embedding_dim) * 0.01
self.W_out = np.random.randn(vocab_size, embedding_dim) * 0.01
def sigmoid(self, x):
"""Sigmoid function with numerical stability."""
return 1 / (1 + np.exp(-np.clip(x, -250, 250)))
def train_step(self, center_idx, context_idx, negative_indices):
"""Perform one training step for a center word and its context."""
# Get embeddings
center_emb = self.W_in[center_idx] # Shape: (embedding_dim,)
context_emb = self.W_out[context_idx] # Shape: (embedding_dim,)
negative_embs = self.W_out[
negative_indices
] # Shape: (negative_samples, embedding_dim)
# Positive example: maximize P(context | center)
pos_score = np.dot(center_emb, context_emb)
pos_prob = self.sigmoid(pos_score)
pos_error = 1 - pos_prob # Target is 1 for positive
# Negative examples: minimize P(negative | center)
neg_scores = np.dot(
center_emb, negative_embs.T
) # Shape: (negative_samples,)
neg_probs = self.sigmoid(neg_scores)
neg_errors = neg_probs # Target is 0 for negative
# Compute gradients
# For input embedding (center word)
grad_in = pos_error * context_emb - np.sum(
neg_errors[:, None] * negative_embs, axis=0
)
# For output embedding (context word)
grad_out_pos = pos_error * center_emb
# For output embeddings (negative words)
grad_out_neg = neg_errors[:, None] * center_emb
# Update weights
self.W_in[center_idx] += self.learning_rate * grad_in
self.W_out[context_idx] += self.learning_rate * grad_out_pos
self.W_out[negative_indices] -= self.learning_rate * grad_out_neg
# Return loss for monitoring
loss = -np.log(pos_prob + 1e-10) - np.sum(np.log(1 - neg_probs + 1e-10))
return loss
def get_embeddings(self):
"""Return the input embeddings as the word representations."""
return self.W_in.copy()
# Example usage with a small vocabulary
vocab = ["cat", "dog", "bird", "car", "plane", "the", "runs", "flies"]
vocab_size = len(vocab)
word_to_idx = {word: i for i, word in enumerate(vocab)}
# Initialize model
model = Word2Vec(
vocab_size=vocab_size,
embedding_dim=10,
window_size=2,
negative_samples=3,
learning_rate=0.1,
)
embedding_shape = model.get_embeddings().shapeInitialized Word2Vec model Embedding shape: (8, 10)
The Word2Vec class implements the skip-gram architecture with negative sampling. The embedding matrices are initialized with small random values, which will be updated during training. The embedding shape shows we have one vector per word in the vocabulary, with each vector having the specified embedding dimension.
Training on Real Data
Now let's train on a larger corpus. We'll use a simple text preprocessing pipeline and train for multiple epochs:
def prepare_training_data(corpus, word_to_idx, window_size=5):
"""Prepare training examples from corpus."""
training_pairs = []
for sentence in corpus:
tokens = sentence.lower().split()
indices = [
word_to_idx.get(word) for word in tokens if word in word_to_idx
]
for i, center_idx in enumerate(indices):
# Get context window
start = max(0, i - window_size)
end = min(len(indices), i + window_size + 1)
for j in range(start, end):
if i != j:
context_idx = indices[j]
training_pairs.append((center_idx, context_idx))
return training_pairs
def sample_negative_words(center_idx, vocab_size, num_samples, word_freqs):
"""Sample negative words weighted by frequency."""
# Create unigram distribution (simplified - in practice use actual frequencies)
# Exclude the center word itself
probs = np.ones(vocab_size)
probs[center_idx] = 0
probs = probs / probs.sum()
negative_indices = np.random.choice(
vocab_size, size=num_samples, replace=False, p=probs
)
return negative_indices
# Prepare corpus
training_corpus = [
"the cat sits on the mat",
"the dog runs in the park",
"the bird flies in the sky",
"the car drives on the road",
"the plane flies in the air",
"the train moves on tracks",
"cats and dogs are pets",
"birds fly in the sky",
"cars drive on roads",
"planes fly in the air",
]
# Build vocabulary from corpus
all_words = []
for sentence in training_corpus:
all_words.extend(sentence.lower().split())
word_counts = Counter(all_words)
vocab = [
word for word, count in word_counts.items() if count >= 1
] # Keep all words for this example
vocab_size = len(vocab)
word_to_idx = {word: i for i, word in enumerate(vocab)}
idx_to_word = {i: word for word, i in word_to_idx.items()}
# Initialize model
model = Word2Vec(
vocab_size=vocab_size,
embedding_dim=20,
window_size=2,
negative_samples=5,
learning_rate=0.1,
)
# Prepare training data
training_pairs = prepare_training_data(
training_corpus, word_to_idx, window_size=2
)
# Train for a few epochs
epochs = 50
loss_history = []
for epoch in range(epochs):
total_loss = 0
random.shuffle(training_pairs)
for center_idx, context_idx in training_pairs:
negative_indices = sample_negative_words(
center_idx, vocab_size, model.negative_samples, None
)
loss = model.train_step(center_idx, context_idx, negative_indices)
total_loss += loss
avg_loss = total_loss / len(training_pairs)
loss_history.append(avg_loss)Vocabulary size: 31 Vocabulary: ['the', 'cat', 'sits', 'on', 'mat', 'dog', 'runs', 'in', 'park', 'bird', 'flies', 'sky', 'car', 'drives', 'road', 'plane', 'air', 'train', 'moves', 'tracks', 'cats', 'and', 'dogs', 'are', 'pets', 'birds', 'fly', 'cars', 'drive', 'roads', 'planes'] Training pairs: 156 Training progress: Epoch 1/50, Average Loss: 4.1583 Epoch 11/50, Average Loss: 2.2776 Epoch 21/50, Average Loss: 1.8914 Epoch 31/50, Average Loss: 1.7206 Epoch 41/50, Average Loss: 1.5997 Training complete! Final loss: 1.5995
The vocabulary contains all unique words from our training corpus. We generate training pairs by sliding a context window across each sentence, creating (center word, context word) pairs. During training, the loss decreases as the model learns to predict context words more accurately. The decreasing loss indicates that the embeddings are learning meaningful relationships: words that appear in similar contexts are being pulled together in embedding space.
Visualizing Training Progress
A loss curve shows how the model improves over training:
The loss curve demonstrates successful training: the loss decreases steadily from the initial random embeddings to a much lower value, indicating the model is learning meaningful word relationships. The initial rapid decrease shows the model quickly learns basic patterns, while the gradual decline and eventual plateau suggest it's converging to a stable solution where embeddings capture semantic relationships effectively.
Visualizing Learned Embeddings
Now let's visualize the embeddings to see if semantically similar words cluster together:
The visualization shows that even with minimal training data, Word2Vec learns meaningful structure. Animals cluster together, vehicles form their own group, and function words occupy a distinct region. This demonstrates how semantic relationships emerge naturally from co-occurrence patterns.
Computing Word Similarities
We can measure similarity between words using cosine similarity:
from sklearn.metrics.pairwise import cosine_similarity
def find_most_similar(word, embeddings, vocab, word_to_idx, top_k=5):
"""Find the most similar words to a given word."""
if word not in word_to_idx:
return []
word_idx = word_to_idx[word]
word_emb = embeddings[
word_idx : word_idx + 1
] # Keep 2D for cosine_similarity
# Compute similarities with all words
similarities = cosine_similarity(word_emb, embeddings)[0]
# Get top k (excluding the word itself)
top_indices = np.argsort(similarities)[::-1][1 : top_k + 1]
results = [(vocab[idx], similarities[idx]) for idx in top_indices]
return results
# Test similarity
embeddings = model.get_embeddings()
cat_similar = find_most_similar("cat", embeddings, vocab, word_to_idx)
car_similar = find_most_similar("car", embeddings, vocab, word_to_idx)Most similar to 'cat': sits: 0.8419 on: 0.7802 mat: 0.7715 road: 0.7640 car: 0.7615 Most similar to 'car': drives: 0.8879 road: 0.8122 mat: 0.8003 cat: 0.7615 on: 0.7345
The similarity scores show how closely related words are in embedding space. Words with higher cosine similarity (closer to 1.0) have more similar embeddings, meaning they appear in similar contexts. For "cat", we expect to see other animals like "dog" or "bird" with high similarity. For "car", we expect vehicles like "plane" or "train". These results demonstrate that Word2Vec successfully captures semantic relationships: words that are semantically related have similar embeddings.
Word Analogies
Word embeddings can capture analogies through vector arithmetic. The classic example is: "king - man + woman = queen". Let's test this:
def word_analogy(word1, word2, word3, embeddings, vocab, word_to_idx, top_k=5):
"""Solve word analogy: word1 is to word2 as word3 is to ?"""
if not all(w in word_to_idx for w in [word1, word2, word3]):
return []
# Compute analogy vector: vec3 + (vec2 - vec1)
vec1 = embeddings[word_to_idx[word1]]
vec2 = embeddings[word_to_idx[word2]]
vec3 = embeddings[word_to_idx[word3]]
analogy_vec = vec3 + (vec2 - vec1)
analogy_vec = analogy_vec.reshape(1, -1)
# Find most similar words
similarities = cosine_similarity(analogy_vec, embeddings)[0]
# Exclude the input words
exclude_indices = [word_to_idx[w] for w in [word1, word2, word3]]
similarities[exclude_indices] = -1
top_indices = np.argsort(similarities)[::-1][:top_k]
results = [(vocab[idx], similarities[idx]) for idx in top_indices]
return results
# Test analogies (with our limited vocabulary)
analogy1 = word_analogy("cat", "dog", "car", embeddings, vocab, word_to_idx)
analogy2 = word_analogy(
"flies", "bird", "drives", embeddings, vocab, word_to_idx
)cat is to dog as car is to ? air: 0.6903 sky: 0.6775 park: 0.6760 runs: 0.6689 planes: 0.5522 flies is to bird as drives is to ? car: 0.8908 cat: 0.7750 road: 0.7588 mat: 0.7574 on: 0.6836
Word analogies test whether embeddings capture relational patterns. The analogy "cat is to dog as car is to ?" asks: what word has the same relationship to "car" that "dog" has to "cat"? We expect "plane" or "train" (another vehicle). The vector arithmetic should point toward the answer. With our small vocabulary and limited training, results may not be perfect, but the principle demonstrates how embeddings encode relationships: semantic relationships become geometric relationships in vector space.
Key Parameters
The Word2Vec implementation uses several key parameters that control training and embedding quality:
-
embedding_dim(default: 100-300): The dimensionality of the embedding vectors. Larger dimensions can capture more nuanced relationships but require more training data and computation. Typical values range from 50-300, with 100-200 being most common for general-purpose embeddings. -
window_size(default: 5-10): The size of the context window on each side of the center word. Larger windows capture more global relationships (document-level patterns), while smaller windows focus on local syntactic relationships. For most applications, 5-10 words on each side works well. -
negative_samples(default: 5-20): The number of negative examples sampled for each positive example. More negative samples improve embedding quality but slow training. Values between 5-20 are typical, with 5-10 being common for large corpora. -
learning_rate(default: 0.01-0.05): The step size for gradient descent updates. Higher learning rates train faster but may overshoot optimal values. Lower learning rates are more stable but require more epochs. Start with 0.01-0.05 and adjust based on loss convergence. -
vocab_size: The number of unique words in the vocabulary. This is determined by your corpus and preprocessing choices. Larger vocabularies require more memory and computation but capture more word diversity.
These parameters interact: larger embedding dimensions and more negative samples improve quality but increase training time. For production use, start with moderate values (embedding_dim=200, window_size=5, negative_samples=5) and tune based on your specific task and computational constraints.