The Distributional Hypothesis › Distributional Similarity

Once we have built a representation of each word as a vector of context counts, we can define word similarity computationally. The intuition is simple: two words are similar if they have similar distributional profiles.

The most widely used measure is cosine similarity, which captures the angle between two vectors regardless of their magnitudes.

$$\text{sim}(\mathbf{u}, \mathbf{v}) = \cos(\theta) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|} = \frac{\sum_{i} u_i v_i}{\sqrt{\sum_i u_i^2} \cdot \sqrt{\sum_i v_i^2}}$$

where:

• $\mathbf{u}$, $\mathbf{v}$: the distributional vectors for two words
• $\mathbf{u} \cdot \mathbf{v}$: their dot product, summing the products of corresponding context counts
• $\|\mathbf{u}\|, \|\mathbf{v}\|$: the Euclidean norms (magnitudes) of each vector
• $\theta$: the angle between the two vectors in the high-dimensional context space

Cosine similarity ranges from $-1$ (pointing in opposite directions) to $+1$ (pointing in the same direction). For non-negative count vectors, values range from $0$ to $1$. A value near $1$ means the words have very similar contextual distributions. A value near $0$ means they appear in almost entirely different contexts.

You might wonder why we use cosine similarity rather than Euclidean distance. The key issue is that raw count vectors are dominated by word frequency. A very common word like "the" will have enormous counts in many dimensions, making its Euclidean distance from other words large regardless of actual semantic similarity. Cosine similarity normalizes by vector length, so it compares the shape of the distributional profile rather than its overall scale. Two words that appear in the same proportional contexts will have high cosine similarity even if one is much more frequent than the other.

To see this concretely, consider two words: "automobile" and "car." Both appear near "drive," "road," "park," and "fuel," but "car" is far more frequent in most corpora. If we use Euclidean distance, the raw count difference dominates. If we use cosine similarity, what matters is whether the proportions of context words are similar, which they are. Cosine similarity correctly identifies these words as closely related.

There is another geometric interpretation. Cosine similarity measures how much two words "point in the same direction" in context space. Each dimension of the vector corresponds to one context word, and the value in that dimension represents how often the target word appears near that context word. If "cat" and "dog" both have large values in the "feed," "pet," and "ran" dimensions, their vectors point in similar directions even if their magnitudes differ. Cosine similarity captures this directional alignment.

Before cosine similarity became dominant, researchers used various other measures of distributional overlap. Jaccard similarity measures the intersection of context sets divided by their union. Dice coefficient is similar but weights the intersection more heavily. These set-based measures work well when you binarize the co-occurrence matrix (recording whether a word appears in a context at all, rather than how many times), but they discard frequency information.

Cosine similarity's advantage is that it naturally handles continuous-valued vectors, including the PMI-weighted vectors we'll cover in the next chapter. Set-based measures require discretization, which loses potentially useful gradient information.

Raw co-occurrence counts have a significant bias: they favor frequent words. "The" co-occurs with almost everything, so raw counts suggest "the" is highly associated with nearly every word in the vocabulary. The next chapter will cover Pointwise Mutual Information (PMI) as a solution to this problem. PMI measures how much more often two words co-occur than expected by chance, which is a more meaningful measure of specific association:

$$\text{PMI}(w, c) = \log \frac{P(w, c)}{P(w) \cdot P(c)}$$

where:

• $P(w, c)$: the probability that words $w$ and $c$ co-occur in a window
• $P(w)$: the marginal probability of seeing word $w$ in any position
• $P(c)$: the marginal probability of seeing word $c$ in any position

A high PMI value means the two words co-occur far more often than you'd expect if they were statistically independent, suggesting a specific semantic relationship. A PMI near zero means their co-occurrence is roughly what chance would predict. For now, know that the choice of what to store in the distributional matrix has a major impact on what similarity computations capture.