The Distributional Hypothesis › The Vector Space Model

The distributional hypothesis gives us a qualitative principle: similar contexts imply similar meanings. The vector space model (VSM) gives this principle a precise mathematical form.

The core idea is to represent each word as a vector in a high-dimensional space where each dimension corresponds to a context word. The value in each dimension records how strongly the target word is associated with that context. Words with similar meanings will have vectors that point in similar directions in this space. Geometric operations on vectors then correspond to semantic operations on words.

Formally, let $V$ be our vocabulary of $|V|$ words. The co-occurrence matrix $M \in \mathbb{R}^{|V| \times |V|}$ has entry:

$$M_{ij} = f(w_i, w_j)$$

where $f(w_i, w_j)$ is some measure of association between word $w_i$ and context word $w_j$. The simplest choice is raw count: $f(w_i, w_j) = \text{count}(w_i, w_j)$, the number of times $w_j$ appears in the context window of $w_i$ across the corpus.

The $i$-th row of $M$, written $\mathbf{m}_i$, is the distributional vector for word $w_i$. This vector lives in $\mathbb{R}^{|V|}$, a space where each of the $|V|$ axes corresponds to one vocabulary word playing the role of context.

The power of this formulation is that it converts a linguistic question (are these two words semantically related?) into a geometric question (do these two vectors point in similar directions?). Geometric operations are well-understood mathematically, can be computed efficiently on modern hardware, and generalize naturally to the continuous representations learned by neural networks.

When we project distributional vectors into two or three dimensions (using PCA or t-SNE), we reliably observe that semantically similar words cluster together. Animals are near other animals. Countries are near other countries. Verbs of motion are near other verbs of motion. This geometric structure is not something we impose; it emerges from the distributional patterns in the text.

What is more striking is that the geometry encodes structured relations. The vector from "king" to "queen" is approximately parallel to the vector from "man" to "woman." The vector from "France" to "Paris" is approximately parallel to the vector from "Germany" to "Berlin." These parallel relationships in the vector space correspond to systematic semantic relationships (gender, capital city) in the world.

This structure arises because paradigmatically related words (words that substitute for each other) have vectors pointing in similar directions, while the systematic differences between related words (the gender difference between "king" and "queen," the country-to-capital mapping) appear as consistent vector offsets. The vector space encodes which words are similar and how they are similar.

This geometric regularity is one of the most striking empirical discoveries in NLP. It suggests that distributional training, despite its simple co-occurrence-counting objective, implicitly discovers the latent semantic structure that humans recognize as meaningful relations. Understanding why this structure emerges is still an active research question, but the fact that it emerges reliably across many corpora and many languages has been thoroughly documented.

A vocabulary of 100,000 words implies a co-occurrence matrix of $10^{10}$ entries. In practice, the vast majority of these entries are zero: most word pairs never co-occur. This extreme sparsity means that storing the full matrix is expensive, and computing cosine similarities between sparse vectors can be slow.

The standard solution is dimensionality reduction. Techniques like Singular Value Decomposition (SVD) find a low-dimensional approximation to the co-occurrence matrix that preserves as much of the variance as possible. Typically, the top 100–300 dimensions capture the most important semantic distinctions, compressing from $|V|$ dimensions to a manageable dense representation.

The key insight is that the semantic structure of language is low-dimensional relative to the vocabulary size. Most of the variation in distributional profiles can be explained by a small number of underlying semantic factors (animacy, concreteness, domain, syntactic category, sentiment). Dimensionality reduction methods discover these underlying factors automatically, without being told what they are.

We'll explore SVD and its relationship to neural word embeddings in later chapters. For now, remember that the transition from sparse count vectors to dense embedded representations is motivated by exactly these considerations: sparsity, dimensionality, and the desire to generalize to contexts not seen in training.