Negative Sampling › Training on a Semantic Corpus
Michael BrenndoerferApril 4, 20256 min read
Language AI HandbookAll chapters Part 4 of 67Word Embeddings
Negative Sampling › Training on a Semantic CorpusOne section · Sections in this chapter10 / 15 ↓
Sections in this chapter15
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's train the model on a corpus with clear semantic structure to observe whether the embeddings capture meaningful relationships:
In[26]:
Codefrom collections import Counter
training_sentences = [
"king queen prince princess royal throne crown palace",
"man woman boy girl child adult person human",
"cat dog pet animal fur paw tail whisker",
"happy sad angry joyful emotion feeling mood cheerful",
"run walk jump sprint move fast slow quick",
"king rules the kingdom with royal power",
"the queen sits on the throne in the palace",
"a man and woman walk with their child",
"the cat and dog are beloved pets",
"feeling happy brings a joyful mood",
"run fast and jump quick to move",
]
all_words = " ".join(training_sentences).lower().split()
vocab = sorted(set(all_words))
word_to_idx = {w: i for i, w in enumerate(vocab)}
idx_to_word = {i: w for w, i in word_to_idx.items()}
word_counts = Counter(all_words)
word_count_by_idx = {word_to_idx[w]: c for w, c in word_counts.items()}
def generate_pairs(sentences, word_to_idx, window=2):
pairs = []
for sentence in sentences:
words = sentence.lower().split()
for i, center in enumerate(words):
for j in range(max(0, i - window), min(len(words), i + window + 1)):
if j != i:
pairs.append((word_to_idx[center], word_to_idx[words[j]]))
return pairs
training_pairs = generate_pairs(training_sentences, word_to_idx, window=2)Out[27]:
ConsoleTraining Corpus Statistics: --------------------------------------------- Number of sentences: 11 Total words: 84 Vocabulary size: 56 Training pairs: 270 Sample vocabulary: ['a', 'adult', 'and', 'angry', 'animal', 'are', 'beloved', 'boy', 'brings', 'cat', 'cheerful', 'child', 'crown', 'dog', 'emotion'] ... and 41 more words
In[28]:
Codenp.random.seed(42)
model = SkipGramNegSampling(
vocab_size=len(vocab),
embedding_dim=30,
word_counts=word_count_by_idx,
num_negatives=5,
alpha=0.75,
)
num_epochs = 50
losses_per_epoch = []
for epoch in range(num_epochs):
np.random.shuffle(training_pairs)
epoch_loss = 0
for center_idx, context_idx in training_pairs:
loss = model.train_pair(center_idx, context_idx, learning_rate=0.1)
epoch_loss += loss
avg_loss = epoch_loss / len(training_pairs)
losses_per_epoch.append(avg_loss)Out[29]:
ConsoleTraining Complete: --------------------------------------------- Epochs: 50 Learning rate: 0.1 Negative samples per pair: 5 Initial loss: 4.1587 Final loss: 1.1451 Reduction: 72.5%
The loss decreases substantially over training, indicating the model is learning to distinguish positive from negative pairs. The percentage reduction reflects how much better the model has become at the binary classification task.
Out[30]:
Visualization
Evaluating the Learned Embeddings
With the model trained, we can examine whether it has captured meaningful semantic relationships:
In[31]:
Codetest_words = ["king", "cat", "happy", "run", "man"]
similarity_results = {}
for word in test_words:
if word in word_to_idx:
similar = model.most_similar(word_to_idx[word], top_n=5)
similarity_results[word] = [
(idx_to_word[idx], sim) for idx, sim in similar
]Out[32]:
ConsoleLearned Word Similarities (Negative Sampling): ------------------------------------------------------- Most similar to 'king': kingdom : +0.759 █████████████ on : +0.709 ████████████ sits : +0.700 ████████████ princess : +0.674 ████████████ queen : +0.647 ████████████ Most similar to 'cat': animal : +0.722 ████████████ are : +0.703 ████████████ sits : +0.594 ███████████ in : +0.582 ███████████ fur : +0.535 ███████████ Most similar to 'happy': joyful : +0.920 ██████████████ emotion : +0.688 ████████████ cheerful : +0.674 ████████████ sad : +0.657 ████████████ brings : +0.643 ████████████ Most similar to 'run': sprint : +0.866 █████████████ quick : +0.760 █████████████ jump : +0.753 █████████████ slow : +0.709 ████████████ to : +0.697 ████████████ Most similar to 'man': girl : +0.831 █████████████ brings : +0.623 ████████████ walk : +0.620 ████████████ boy : +0.617 ████████████ and : +0.563 ███████████
Words from the same semantic category cluster together. "King" finds other royalty terms, "cat" finds animal terms, and "happy" finds emotion words. This emerges purely from the binary classification objective applied to co-occurrence pairs.
Out[33]:
Visualization