The Distributional Hypothesis › Implementation: Computing Distributional Similarity
Language AI HandbookAll chapters The Distributional Hypothesis › Implementation: Computing Distributional SimilarityOne section · Sections in this chapter7 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement the full pipeline from raw text to distributional similarity from scratch. This will make every step of the hypothesis concrete.
We start by building a small but illustrative corpus of sentences. The choice of words is designed so that semantically related words ("cat"/"dog", "run"/"walk", "apple"/"banana") should end up with high cosine similarities after the distributional computation.
# A small corpus where semantically similar words share contexts
corpus = [
"the cat sat on the mat",
"the dog sat on the floor",
"the cat ran across the yard",
"the dog ran across the yard",
"i saw a cat near the tree",
"i saw a dog near the tree",
"she fed the cat some fish",
"she fed the dog some meat",
"the apple fell from the tree",
"the banana fell from the tree",
"i ate an apple for breakfast",
"i ate a banana for breakfast",
"he likes to run every morning",
"she likes to walk every morning",
"they decided to run in the park",
"they decided to walk in the park",
]
# Tokenize and build vocabulary
all_tokens = []
for sentence in corpus:
all_tokens.extend(sentence.lower().split())
vocab = sorted(set(all_tokens))
word_to_idx = {word: i for i, word in enumerate(vocab)}
idx_to_word = {i: word for word, i in word_to_idx.items()}
vocab_size = len(vocab)Vocabulary size: 39 Vocabulary: ['a', 'across', 'an', 'apple', 'ate', 'banana', 'breakfast', 'cat', 'decided', 'dog', 'every', 'fed', 'fell', 'fish', 'floor', 'for', 'from', 'he', 'i', 'in', 'likes', 'mat', 'meat', 'morning', 'near', 'on', 'park', 'ran', 'run', 'sat', 'saw', 'she', 'some', 'the', 'they', 'to', 'tree', 'walk', 'yard']
Our vocabulary has a manageable set of words that lets us verify results by inspection. Now let's build the co-occurrence matrix with a window of size 2.
import numpy as np
def build_cooccurrence_matrix(corpus, word_to_idx, window_size=2):
"""Build a symmetric word-word co-occurrence matrix."""
V = len(word_to_idx)
matrix = np.zeros((V, V))
for sentence in corpus:
tokens = sentence.lower().split()
for i, word in enumerate(tokens):
if word not in word_to_idx:
continue
w_idx = word_to_idx[word]
left = max(0, i - window_size)
right = min(len(tokens), i + window_size + 1)
for j in range(left, right):
if j == i:
continue
context_word = tokens[j]
if context_word not in word_to_idx:
continue
c_idx = word_to_idx[context_word]
matrix[w_idx, c_idx] += 1
return matrix
cooccurrence = build_cooccurrence_matrix(corpus, word_to_idx, window_size=2)Co-occurrence matrix shape: (39, 39) Co-occurrence counts for 'cat': 'the': 4 'a': 1 'across': 1 'fed': 1 'fish': 1 'near': 1 'on': 1 'ran': 1 'sat': 1 'saw': 1 'some': 1
The context words for "cat" reveal its distributional profile: function words like "the" appear frequently simply because they appear everywhere, while content words like "dog," "ran," "sat" reflect specific semantic associations. This is the frequency bias in action, and it motivates the PMI weighting we'll explore in the next chapter.
Now we compute cosine similarity between any two words.
def cosine_similarity(vec1, vec2):
"""Compute cosine similarity between two vectors."""
dot = np.dot(vec1, vec2)
norm1 = np.linalg.norm(vec1)
norm2 = np.linalg.norm(vec2)
if norm1 == 0 or norm2 == 0:
return 0.0
return dot / (norm1 * norm2)
def word_similarity(word1, word2, matrix, word_to_idx):
"""Return cosine similarity between two words' distributional vectors."""
if word1 not in word_to_idx or word2 not in word_to_idx:
raise ValueError("Word not in vocabulary")
vec1 = matrix[word_to_idx[word1]]
vec2 = matrix[word_to_idx[word2]]
return cosine_similarity(vec1, vec2)Word similarity scores (cosine similarity): Word Pair Cosine Similarity ----------------------------------------------- 'cat' vs 'dog': 0.9615 'run' vs 'walk': 1.0000 'apple' vs 'banana': 0.8571 'cat' vs 'apple': 0.2965 'run' vs 'apple': 0.1195 'the' vs 'a': 0.2326
The results confirm the distributional hypothesis in action. "Cat" and "dog" have high similarity because they appear near the same words: "the," "sat," "ran," "fed," "saw," and "tree." "Run" and "walk" score high because they share sentence frames: "likes to [run/walk] every morning" and "decided to [run/walk] in the park." "Apple" and "banana" are similar for the same reason. Meanwhile, "cat" and "apple" score much lower because their contexts barely overlap.
The result for "the" and "a" is also interesting. Both are determiners that can appear before any noun, so their contexts overlap substantially. This is an example of paradigmatic similarity among function words, an observed linguistic pattern, even though it does not reflect semantic relatedness in the usual sense.
Most Similar Words
A natural use of distributional similarity is finding the nearest neighbors of any word.
def most_similar(target_word, matrix, word_to_idx, idx_to_word, top_n=5):
"""Find the most similar words to a target based on cosine similarity."""
if target_word not in word_to_idx:
raise ValueError(f"'{target_word}' not in vocabulary")
target_vec = matrix[word_to_idx[target_word]]
similarities = []
for word, idx in word_to_idx.items():
if word == target_word:
continue
vec = matrix[idx]
sim = cosine_similarity(target_vec, vec)
similarities.append((word, sim))
similarities.sort(key=lambda x: -x[1])
return similarities[:top_n]Most similar to 'cat': 'dog': 0.9615 'ran': 0.7526 'sat': 0.7526 'tree': 0.7206 'floor': 0.6934 Most similar to 'run': 'walk': 1.0000 'he': 0.6708 'they': 0.6708 'every': 0.6000 'in': 0.5071 Most similar to 'apple': 'banana': 0.8571 'fell': 0.4835 'tree': 0.4629 'from': 0.4041 'for': 0.4009
The nearest neighbors of "cat" are other animate nouns and the verbs that describe their behavior. The nearest neighbors of "run" are words from similar activity frames. This is the distributional hypothesis working: without any pre-programmed knowledge about what words mean, the system has discovered semantic groupings purely from co-occurrence patterns.
A Worked Example: Tracing the Similarity Computation
To make the computation fully transparent, let's trace exactly what happens when we compute the similarity between "cat" and "dog."
# Inspect the distributional vectors for cat and dog side by side
cat_vec = cooccurrence[word_to_idx["cat"]]
dog_vec = cooccurrence[word_to_idx["dog"]]
# Find dimensions where both words have nonzero counts
shared_dims = [
(i, cat_vec[i], dog_vec[i])
for i in range(vocab_size)
if cat_vec[i] > 0 and dog_vec[i] > 0
]
shared_dims.sort(key=lambda x: -(x[1] + x[2]))Shared context dimensions for 'cat' and 'dog': Context word cat count dog count ---------------------------------------- the 4 4 a 1 1 across 1 1 fed 1 1 near 1 1 on 1 1 ran 1 1 sat 1 1 saw 1 1 some 1 1 Dot product: 25.0 ||cat||: 5.0990 ||dog||: 5.0990 Cosine similarity: 0.9615
This trace shows exactly which shared contexts drive the similarity score. The words "the," "sat," "ran," "across," "yard," "saw," "a," "near," "tree," "fed," and "some" all appear in both "cat" and "dog" contexts. The dot product sums the products of their counts in each of these shared dimensions, and dividing by the product of the norms normalizes for the fact that both words have different total frequencies.