Word Embeddings: Word2Vec, GloVe and Vector Semantics › Worked Example: Understanding Embedding Space
Word Embeddings: Word2Vec, GloVe and Vector Semantics › Worked Example: Understanding Embedding SpaceOne section · Sections in this article5 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's work through a concrete example to see how embeddings capture semantic relationships. We'll use a small vocabulary and train simple embeddings to illustrate the concepts.
Setting Up a Mini Corpus
Consider this tiny corpus about animals and transportation:
corpus = [
"the cat sits on the mat",
"the dog runs in the park",
"the bird flies in the sky",
"the car drives on the road",
"the plane flies in the air",
"the train moves on tracks",
]Even with this minimal data, we can see patterns: "cat" and "dog" appear in similar contexts (both with "the", both animals), while "car" and "plane" share "flies" or movement verbs but differ in their specific contexts.
Computing Co-occurrence Statistics
Let's build a co-occurrence matrix to understand what GloVe would learn:
from collections import defaultdict
import numpy as np
def build_cooccurrence_matrix(corpus, window_size=2):
"""Build a co-occurrence matrix from a corpus."""
# Tokenize
sentences = [sentence.split() for sentence in corpus]
# Build vocabulary
vocab = set()
for sentence in sentences:
vocab.update(sentence)
vocab = sorted(list(vocab))
word_to_idx = {word: i for i, word in enumerate(vocab)}
# Initialize co-occurrence matrix
cooccurrence = defaultdict(int)
# Count co-occurrences
for sentence in sentences:
for i, center_word in enumerate(sentence):
center_idx = word_to_idx[center_word]
# Look at context window
for j in range(
max(0, i - window_size), min(len(sentence), i + window_size + 1)
):
if i != j:
context_word = sentence[j]
context_idx = word_to_idx[context_word]
cooccurrence[(center_idx, context_idx)] += 1
# Convert to matrix
V = len(vocab)
matrix = np.zeros((V, V))
for (i, j), count in cooccurrence.items():
matrix[i, j] = count
return matrix, vocab, word_to_idx
cooccurrence_matrix, vocabulary, word_to_idx = build_cooccurrence_matrix(corpus)Vocabulary: ['air', 'bird', 'car', 'cat', 'dog', 'drives', 'flies', 'in', 'mat', 'moves', 'on', 'park', 'plane', 'road', 'runs', 'sits', 'sky', 'the', 'tracks', 'train'] Vocabulary size: 20 Co-occurrence matrix (first 10x10): [[0. 0. 0. 0. 0. 0. 0. 1. 0. 0.] [0. 0. 0. 0. 0. 0. 1. 1. 0. 0.] [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 1. 0. 0.] [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.] [0. 1. 0. 0. 0. 0. 0. 2. 0. 0.] [1. 1. 0. 0. 1. 0. 2. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
The vocabulary contains all unique words from our small corpus. The co-occurrence matrix shows how often each word pair appears together within the context window. Notice that words like "cat" and "dog" have similar co-occurrence patterns (both frequently co-occur with "the"), while "car" and "plane" share some patterns but differ in others. This is exactly what embeddings learn to capture: words with similar co-occurrence patterns will have similar embeddings.
Visualizing the Co-occurrence Matrix
A heatmap visualization makes the co-occurrence patterns immediately visible:
The heatmap reveals several important patterns. First, the matrix is sparse: most word pairs never co-occur (white cells with count 0). Second, semantically related words show similar patterns: "cat" and "dog" both have high co-occurrence with "the", "on", and "in", because they play similar grammatical roles. Third, the diagonal is empty because we don't count a word co-occurring with itself. These patterns are exactly what GloVe's matrix factorization captures: words with similar rows (or columns) in this matrix will have similar embeddings after training.
Visualizing Relationships
Even in this tiny example, we can see semantic clusters emerging. Words that appear in similar contexts will have similar rows (or columns) in the co-occurrence matrix, which translates to similar embeddings after factorization.