The Distributional Hypothesis: Word Meaning from Context

What does a word mean? A philosopher might say a word refers to a concept in the mind. A logician might point to truth conditions, the set of objects in the world to which the word correctly applies. But a linguist named John Rupert Firth had a different answer, one that would eventually reshape how computers process language.

Firth's famous observation, stated in 1957, was simple: "You shall know a word by the company it keeps." The idea is that meaning is not some abstract property a word carries around by itself. Meaning emerges from patterns of use. Words that appear in the same kinds of sentences, surrounded by the same kinds of neighbors, tend to mean similar things. This is the distributional hypothesis, and it is the theoretical foundation for nearly everything in modern NLP, from bag-of-words models to word2vec to transformer language models.

This chapter unpacks what the distributional hypothesis says, why it holds, and what it enables. We'll look at context windows, the distinction between paradigmatic and syntagmatic word relations, and how to compute word similarity directly from distributional evidence. We'll also examine its limitations directly, because understanding where a theory breaks down is just as important as understanding where it works.

Before formalizing anything, let's build intuition. Suppose you encounter the word "wug" in these sentences:

• "The wug ran across the field."
• "She fed the wug a carrot."
• "The wug's fur was soft and brown."
• "A wug can run up to 20 miles per hour."

You've never seen the word before, but you already know quite a lot about it. A wug is an animal. It is probably medium-sized and furry, probably also fast. It eats carrots, which hints at an herbivore. From pure context, with zero dictionary access, you've inferred a rich meaning.

This is what the distributional hypothesis says is happening all the time, with every word, in every language. When a child learns that "dog" and "cat" are both things you feed and pet and that run and play, they are building a distributional profile. When a reader encounters a technical term they've never defined, they infer meaning from the sentences it lives in. When a language model predicts what comes next, it is, in a very real sense, using context to represent meaning.

The distributional hypothesis does not claim that context defines meaning completely. Rather, it makes a weaker and more useful claim: context provides reliable evidence about meaning. Two words with highly overlapping contexts are almost certainly semantically related. This gives us a computational toehold. If we can measure context, we can measure similarity, and similarity is an enormous part of what meaning involves.

The hypothesis also has a deep empirical grounding. Across many languages and many corpora, researchers have confirmed that distributional similarity correlates strongly with human judgments of semantic relatedness. When you ask people to rate how similar "cat" and "dog" are versus "cat" and "bicycle," the distributional estimates track those human intuitions remarkably well. This is not a given. Many things that are logically elegant fail to match empirical data. The fact that distributional profiles correlate with human meaning judgments is what transformed an interesting linguistic observation into a practical engineering tool.

It's worth pausing to ask why this principle holds at all. Why would the contexts a word appears in be informative about its meaning?

A practical answer is that language is a social, communicative activity. Speakers choose words based on what they're trying to convey, and they place those words in sentences that make their meaning recoverable for a listener or reader. If two words can be used in the same communicative contexts, it is because they can serve the same communicative purposes, and that functional equivalence is deeply related to semantic equivalence.

There is also a recursive dynamic at work. The meaning of "dog" is partly determined by the contexts it appears in, but those contexts are themselves composed of other words whose meanings are similarly determined. "Dog" appears near "bark," "leash," "collar," and "fetch," and those words in turn appear near other semantically coherent clusters. The entire lexicon forms a web of mutual constraints, and the distributional hypothesis says we can discover this web by counting co-occurrences.

A third perspective comes from cognitive science. Psycholinguistic studies have shown that when people read sentences, they activate the meanings of words they haven't seen yet based purely on prior context. When you read "She reached into the refrigerator and pulled out a cold..." your brain has already predicted that the next word is likely a food item, probably something you'd store cold. Your mental model of what words can appear in what positions predicts meaning. The distributional hypothesis is essentially a corpus-level formalization of this prediction process.

Zellig Harris, a structural linguist writing in the 1950s, articulated the formal version of this principle first. Harris observed that morphological and syntactic regularities could be discovered by examining distributions of phonemes and words without reference to meaning at all. He called this the "distributional method," and it was meant as a technique for linguistic analysis, not as a theory of semantics.

Firth, working in a different tradition rooted in British contextualism, was more interested in meaning than in structure. His contribution was to take Harris's distributional insight and redirect it toward semantics. His claim went further than treating distributional patterns as useful evidence for linguistic analysis: he argued that they constitute meaning. Words acquire their meanings through the habitual company they keep. A word like "bank" means different things partly because of its different distributional contexts: financial sentences versus river sentences.

These ideas sat largely dormant in computational form until the 1980s and 1990s, when the availability of large digital corpora made it practical to compute distributions at scale. The Brown Corpus, released in 1961, was among the first systematic collections of machine-readable text. By the 1980s, researchers were building gigabyte-scale corpora from newspaper archives. Researchers including Church and Hanks, Turney, and Schütze began developing mathematical frameworks for measuring distributional similarity, laying the groundwork for what we now call word vectors.

Church and Hanks's 1990 paper on Pointwise Mutual Information (PMI) was particularly influential. They showed that co-occurrence counts, properly normalized, could capture specific lexical associations rather than just frequency artifacts. Schütze's work on context vectors in the early 1990s demonstrated that high-dimensional co-occurrence vectors could be compressed into low-dimensional representations that preserved semantic structure. These ideas directly anticipate word2vec, GloVe, and ultimately the attention-based context representations of transformers.

To operationalize the distributional hypothesis, we need a precise definition of "context." The most common approach uses a context window: a region of text surrounding the target word within which we count co-occurring words.

A window of size $k$ centered on target word $w_i$ includes all words within $k$ positions to the left or right:

where:

• $w_i$: the target word at position $i$ in the corpus
• $k$: the half-width of the context window (positions counted from the target)
• $w_{i-k}, \ldots, w_{i-1}$: the $k$ words to the left of the target
• $w_{i+1}, \ldots, w_{i+k}$: the $k$ words to the right of the target

For example, in the sentence "The quick brown fox jumps over the lazy dog," with $k = 2$ and target word "fox":

• Left context: {"quick", "brown"}
• Right context: {"jumps", "over"}

The choice of $k$ matters enormously and captures different kinds of linguistic information. Small windows (1 to 3 words) capture tight syntactic and local semantic patterns. "Fast" and "rapid" tend to appear before the same nouns, and a tight window catches this. Large windows (5 to 10 or more words) capture topical associations. "Doctor" and "hospital" may not appear within 2 words of each other, but they frequently appear in the same broader region of text.

The window size is best understood not as a binary choice but as a dial that controls what kind of semantic relationship the model emphasizes. The right window size depends on what downstream task you care about. For lexical substitution, "synonyms that can replace each other in a sentence," small windows work better. For topic modeling, "words that appear together in discussions of the same subject," larger windows or document-level co-occurrence is more appropriate.

A practical question is whether the context window should cross sentence boundaries. Most implementations do not let windows span beyond a sentence, because linguistic relationships primarily hold within sentences. A word near the end of one sentence is not semantically related to a word near the beginning of the next just because they happen to be nearby in the raw text stream.

This boundary enforcement also prevents spurious associations. If two unrelated sentences happen to be adjacent in a corpus, we don't want their terminal and initial words to be artificially linked. Respecting sentence boundaries keeps the co-occurrence evidence linguistically grounded.

There is one case where cross-sentence context is sometimes permitted: when modeling discourse coherence or long-range topical associations. In these cases, document-level co-occurrence (counting whether two words appear anywhere in the same document, regardless of position) is often more appropriate than a window that straddles sentence endings.

One refinement is to give closer context words more weight than distant ones. A word appearing immediately next to the target is more likely to be semantically relevant than one five positions away. A distance-weighted count for position offset $\delta$ might use a weight of $1/|\delta|$, giving:

where:

• $w_i$: the target word
• $c$: a candidate context word
• $\delta$: the signed offset from the target position
• $\mathbf{1}[w_{i+\delta} = c]$: an indicator that equals 1 if the word at offset $\delta$ is $c$, and 0 otherwise
• $\frac{1}{|\delta|}$: the inverse-distance weight, higher for closer positions

This weighting is used in some early word vector methods and can improve the quality of similarity estimates. Word2vec's skip-gram model, rather than using explicit weighting, effectively achieves a similar result by training on randomly sampled context words, which under uniform sampling produces context distributions concentrated around the target.

The standard context window is symmetric: it extends equally to both sides of the target. But some distributional models use asymmetric windows, considering only left-context or right-context separately. Left and right contexts encode different kinds of information. In English, the left context of a noun often contains adjectives and determiners, while the right context contains verbs and prepositions. Separating these can capture finer-grained syntactic information.

Some models build separate left-context and right-context vectors and concatenate them, giving a richer representation that encodes both what comes before a word and what comes after. This directional approach is particularly useful for tasks that require syntactic sensitivity.

The distributional hypothesis gives us a qualitative principle: similar contexts imply similar meanings. The vector space model (VSM) gives this principle a precise mathematical form.

The core idea is to represent each word as a vector in a high-dimensional space where each dimension corresponds to a context word. The value in each dimension records how strongly the target word is associated with that context. Words with similar meanings will have vectors that point in similar directions in this space. Geometric operations on vectors then correspond to semantic operations on words.

Formally, let $V$ be our vocabulary of $|V|$ words. The co-occurrence matrix $M \in \mathbb{R}^{|V| \times |V|}$ has entry:

where $f(w_i, w_j)$ is some measure of association between word $w_i$ and context word $w_j$. The simplest choice is raw count: $f(w_i, w_j) = \text{count}(w_i, w_j)$, the number of times $w_j$ appears in the context window of $w_i$ across the corpus.

The $i$-th row of $M$, written $\mathbf{m}_i$, is the distributional vector for word $w_i$. This vector lives in $\mathbb{R}^{|V|}$, a space where each of the $|V|$ axes corresponds to one vocabulary word playing the role of context.

The power of this formulation is that it converts a linguistic question (are these two words semantically related?) into a geometric question (do these two vectors point in similar directions?). Geometric operations are well-understood mathematically, can be computed efficiently on modern hardware, and generalize naturally to the continuous representations learned by neural networks.

When we project distributional vectors into two or three dimensions (using PCA or t-SNE), we reliably observe that semantically similar words cluster together. Animals are near other animals. Countries are near other countries. Verbs of motion are near other verbs of motion. This geometric structure is not something we impose; it emerges from the distributional patterns in the text.

What is more striking is that the geometry encodes structured relations. The vector from "king" to "queen" is approximately parallel to the vector from "man" to "woman." The vector from "France" to "Paris" is approximately parallel to the vector from "Germany" to "Berlin." These parallel relationships in the vector space correspond to systematic semantic relationships (gender, capital city) in the world.

This structure arises because paradigmatically related words (words that substitute for each other) have vectors pointing in similar directions, while the systematic differences between related words (the gender difference between "king" and "queen," the country-to-capital mapping) appear as consistent vector offsets. The vector space encodes which words are similar and how they are similar.

This geometric regularity is one of the most striking empirical discoveries in NLP. It suggests that distributional training, despite its simple co-occurrence-counting objective, implicitly discovers the latent semantic structure that humans recognize as meaningful relations. Understanding why this structure emerges is still an active research question, but the fact that it emerges reliably across many corpora and many languages has been thoroughly documented.

A vocabulary of 100,000 words implies a co-occurrence matrix of $10^{10}$ entries. In practice, the vast majority of these entries are zero: most word pairs never co-occur. This extreme sparsity means that storing the full matrix is expensive, and computing cosine similarities between sparse vectors can be slow.

The standard solution is dimensionality reduction. Techniques like Singular Value Decomposition (SVD) find a low-dimensional approximation to the co-occurrence matrix that preserves as much of the variance as possible. Typically, the top 100–300 dimensions capture the most important semantic distinctions, compressing from $|V|$ dimensions to a manageable dense representation.

The key insight is that the semantic structure of language is low-dimensional relative to the vocabulary size. Most of the variation in distributional profiles can be explained by a small number of underlying semantic factors (animacy, concreteness, domain, syntactic category, sentiment). Dimensionality reduction methods discover these underlying factors automatically, without being told what they are.

We'll explore SVD and its relationship to neural word embeddings in later chapters. For now, remember that the transition from sparse count vectors to dense embedded representations is motivated by exactly these considerations: sparsity, dimensionality, and the desire to generalize to contexts not seen in training.

One of the most important distinctions in distributional semantics is between two types of word relationships captured by context.

Syntagmatic relations hold between words that commonly appear together, in sequence. "Bread" and "butter," "coffee" and "hot," "kick" and "ball," are syntagmatically related. These words tend to co-occur in the same context windows. If you build a distributional representation that counts direct co-occurrence, syntagmatic relations will be prominent.

Paradigmatic relations hold between words that appear in the same positions relative to other words, but not necessarily together. "Dog" and "cat" are paradigmatically related: both can follow "the," both can be the subject of "ran" or "sat," both appear near "pet" and "feed." These words occupy the same functional slot in sentences. They are substitutable for each other.

Whether a distributional model captures syntagmatic or paradigmatic relations depends heavily on the context window size and the type of co-occurrence counted.

• Small windows and direct co-occurrence counts emphasize syntagmatic relations. Words that appear next to each other will have high overlap.
• Larger windows and especially document-level co-occurrence emphasize paradigmatic (topical) relations. "King" and "queen" both appear in documents about royalty, so their document-level distributions overlap.

This distinction matters practically because different NLP tasks call for different types of similarity. Thesaurus construction requires paradigmatic similarity: you want to find words that mean the same thing, which means they must be substitutable in the same positions. Information retrieval often benefits from syntagmatic similarity: knowing that "search" and "query" appear near similar words helps match a user query to relevant documents.

Word2vec's skip-gram model, which we'll explore in later chapters, was specifically designed with small windows to capture paradigmatic similarity. The classic result that "king - man + woman $\approx$ queen" works because the model learned paradigmatic functional relationships. "King" and "queen" appear in similar syntactic positions (subject of sentences about royalty, object of "crowned"), which means their distributional vectors are similar, and the arithmetic operation captures the gender-role component of that similarity.

Related to the paradigmatic/syntagmatic distinction is the difference between thematic and taxonomic similarity. Thematic similarity groups words that appear together in the same event or scene: "coffee," "mug," "hot," and "morning" are thematically related because they often co-occur in descriptions of the same activity. Taxonomic similarity groups words of the same type: "coffee," "tea," "juice," and "water" are all beverages.

Distributional methods tend to capture both, but the balance shifts with window size and context definition. Tight windows in rich syntactic contexts often capture taxonomic similarity better, because words of the same syntactic category (nouns, verbs, adjectives) appear in structurally similar positions. Looser, topic-oriented windows capture thematic similarity better.

Human semantic memory appears to contain both types of relations, organized in overlapping networks. The distributional hypothesis provides a single framework that, with the right settings, can extract either type from raw text.

Once we have built a representation of each word as a vector of context counts, we can define word similarity computationally. The intuition is simple: two words are similar if they have similar distributional profiles.

The most widely used measure is cosine similarity, which captures the angle between two vectors regardless of their magnitudes.

where:

• $\mathbf{u}$, $\mathbf{v}$: the distributional vectors for two words
• $\mathbf{u} \cdot \mathbf{v}$: their dot product, summing the products of corresponding context counts
• $\|\mathbf{u}\|, \|\mathbf{v}\|$: the Euclidean norms (magnitudes) of each vector
• $\theta$: the angle between the two vectors in the high-dimensional context space

Cosine similarity ranges from $-1$ (pointing in opposite directions) to $+1$ (pointing in the same direction). For non-negative count vectors, values range from $0$ to $1$. A value near $1$ means the words have very similar contextual distributions. A value near $0$ means they appear in almost entirely different contexts.

You might wonder why we use cosine similarity rather than Euclidean distance. The key issue is that raw count vectors are dominated by word frequency. A very common word like "the" will have enormous counts in many dimensions, making its Euclidean distance from other words large regardless of actual semantic similarity. Cosine similarity normalizes by vector length, so it compares the shape of the distributional profile rather than its overall scale. Two words that appear in the same proportional contexts will have high cosine similarity even if one is much more frequent than the other.

To see this concretely, consider two words: "automobile" and "car." Both appear near "drive," "road," "park," and "fuel," but "car" is far more frequent in most corpora. If we use Euclidean distance, the raw count difference dominates. If we use cosine similarity, what matters is whether the proportions of context words are similar, which they are. Cosine similarity correctly identifies these words as closely related.

There is another geometric interpretation. Cosine similarity measures how much two words "point in the same direction" in context space. Each dimension of the vector corresponds to one context word, and the value in that dimension represents how often the target word appears near that context word. If "cat" and "dog" both have large values in the "feed," "pet," and "ran" dimensions, their vectors point in similar directions even if their magnitudes differ. Cosine similarity captures this directional alignment.

Before cosine similarity became dominant, researchers used various other measures of distributional overlap. Jaccard similarity measures the intersection of context sets divided by their union. Dice coefficient is similar but weights the intersection more heavily. These set-based measures work well when you binarize the co-occurrence matrix (recording whether a word appears in a context at all, rather than how many times), but they discard frequency information.

Cosine similarity's advantage is that it naturally handles continuous-valued vectors, including the PMI-weighted vectors we'll cover in the next chapter. Set-based measures require discretization, which loses potentially useful gradient information.

Raw co-occurrence counts have a significant bias: they favor frequent words. "The" co-occurs with almost everything, so raw counts suggest "the" is highly associated with nearly every word in the vocabulary. The next chapter will cover Pointwise Mutual Information (PMI) as a solution to this problem. PMI measures how much more often two words co-occur than expected by chance, which is a more meaningful measure of specific association:

where:

• $P(w, c)$: the probability that words $w$ and $c$ co-occur in a window
• $P(w)$: the marginal probability of seeing word $w$ in any position
• $P(c)$: the marginal probability of seeing word $c$ in any position

A high PMI value means the two words co-occur far more often than you'd expect if they were statistically independent, suggesting a specific semantic relationship. A PMI near zero means their co-occurrence is roughly what chance would predict. For now, know that the choice of what to store in the distributional matrix has a major impact on what similarity computations capture.

Let's implement the full pipeline from raw text to distributional similarity from scratch. This will make every step of the hypothesis concrete.

We start by building a small but illustrative corpus of sentences. The choice of words is designed so that semantically related words ("cat"/"dog", "run"/"walk", "apple"/"banana") should end up with high cosine similarities after the distributional computation.

The results confirm the distributional hypothesis in action. "Cat" and "dog" have high similarity because they appear near the same words: "the," "sat," "ran," "fed," "saw," and "tree." "Run" and "walk" score high because they share sentence frames: "likes to [run/walk] every morning" and "decided to [run/walk] in the park." "Apple" and "banana" are similar for the same reason. Meanwhile, "cat" and "apple" score much lower because their contexts barely overlap.

The result for "the" and "a" is also interesting. Both are determiners that can appear before any noun, so their contexts overlap substantially. This is an example of paradigmatic similarity among function words, an observed linguistic pattern, even though it does not reflect semantic relatedness in the usual sense.

A natural use of distributional similarity is finding the nearest neighbors of any word.

The nearest neighbors of "cat" are other animate nouns and the verbs that describe their behavior. The nearest neighbors of "run" are words from similar activity frames. This is the distributional hypothesis working: without any pre-programmed knowledge about what words mean, the system has discovered semantic groupings purely from co-occurrence patterns.

To make the computation fully transparent, let's trace exactly what happens when we compute the similarity between "cat" and "dog."

This trace shows exactly which shared contexts drive the similarity score. The words "the," "sat," "ran," "across," "yard," "saw," "a," "near," "tree," "fed," and "some" all appear in both "cat" and "dog" contexts. The dot product sums the products of their counts in each of these shared dimensions, and dividing by the product of the norms normalizes for the fact that both words have different total frequencies.

Let's visualize the distributional structure we've computed. A 2D projection makes the similarity structure intuitive.

The PCA projection shows the distributional structure. Animals ("cat," "dog") cluster together, as do fruits ("apple," "banana") and motion verbs ("run," "walk"). These groupings emerge from no source other than co-occurrence patterns in our small corpus. This is a scaled-down but faithful demonstration of what large-scale distributional methods achieve.

The fact that PCA, a purely linear transformation, can separate these semantic groups is significant. It means the distributional signal is strong enough to be captured in the first two principal components of variance, the directions along which the word vectors differ most from one another. In a real corpus with millions of sentences, this structure becomes even cleaner.

The choice of window size is a model hyperparameter that affects what kind of similarity the system learns. Let's visualize how similarity estimates change as we vary window size.

Semantically related pairs like "cat"/"dog" and "run"/"walk" generally score high across all window sizes because their shared contexts appear at both close and distant offsets. Unrelated pairs like "cat"/"apple" and "run"/"banana" remain low. The window size primarily affects how much overlap accumulates in the middle range, not which pairs are fundamentally similar.

This stability across window sizes in a small corpus is partly due to our corpus design: we deliberately wrote parallel sentences. In real corpora, the relationship between window size and captured similarity type is more pronounced, and choosing the right window size for your task matters.

It is worth stepping back and thinking about what we have built. A co-occurrence matrix is a $|V| \times |V|$ matrix where $|V|$ is the vocabulary size. The entry at row $i$, column $j$ records how often word $i$ appears in the context of word $j$.

where $\text{count}(w_i, w_j, k)$ is the number of times word $w_j$ appears within a window of size $k$ around word $w_i$ across the entire corpus.

This matrix is the distributional representation of the entire vocabulary. Each row is the distributional vector for one word. The columns represent context dimensions. The whole matrix is the computational realization of the distributional hypothesis.

This representation has some striking properties:

• Symmetry: If we use a symmetric context window, $M_{ij} = M_{ji}$ because every time word $j$ appears in the context of word $i$, word $i$ also appears in the context of word $j$. The matrix is symmetric.
• Sparsity: In any real vocabulary, most word pairs never co-occur. A vocabulary of 100,000 words would require a $10^{10}$-entry matrix, the vast majority of whose entries are zero. This sparsity is one motivation for the dimensionality reduction techniques covered in later chapters.
• High dimensionality: Even for our toy corpus, the matrix has thousands of entries. For real corpora, the vocabulary might have hundreds of thousands of words, making the full matrix computationally expensive to store and compare.
• Frequency bias: The matrix directly reflects word frequencies. Common words like "the" have large counts everywhere, distorting similarity estimates toward high-frequency vocabulary.

The progression from raw co-occurrence matrices to PMI-weighted matrices to dense word embeddings (word2vec, GloVe) is the story of how the field addressed these limitations while preserving the core distributional insight.

A legitimate question when building any model is: how do we know if it is working? For distributional models, evaluation typically proceeds along two axes: intrinsic evaluation, which measures the quality of the representations themselves, and extrinsic evaluation, which measures performance on downstream tasks.

The most direct form of intrinsic evaluation compares distributional similarity scores to human judgment datasets. Researchers have collected datasets where pairs of words are rated by many human annotators for their semantic similarity or relatedness, on a scale from 0 (completely unrelated) to 10 (identical meaning). Classic benchmarks include:

• WordSim-353: 353 word pairs rated for similarity and relatedness. The dataset conflates two distinct human judgments (semantic similarity and general relatedness), which makes it somewhat ambiguous but historically important.
• SimLex-999: 999 word pairs with human ratings specifically focused on similarity (words meaning the same thing) rather than association (words that appear together). SimLex-999 is harder for distributional models because it penalizes conflating antonyms with synonyms.
• MEN: 3,000 word pairs rated for semantic relatedness, drawn from a broader vocabulary.
• RG-65: An older but widely-cited dataset of 65 noun pairs with human similarity ratings.

For each dataset, the evaluation computes the correlation (usually Spearman rank correlation) between the distributional cosine similarities and the human ratings. A perfect distributional model would assign high similarity exactly to the pairs humans rate as similar, and low similarity to pairs humans rate as dissimilar.

A standard count-based distributional model achieves Spearman correlations around 0.65 to 0.75 on WordSim-353 after PMI weighting. Neural embedding methods like word2vec and GloVe achieve 0.70 to 0.80. Contextual models like BERT, using representations from the model's final layers, achieve higher scores still. These numbers give a rough calibration: even the best models do not perfectly replicate human similarity judgments, and there is clear headroom for improvement.

The specific choice of Spearman (rank-order) correlation rather than Pearson (linear) correlation is worth explaining. We use Spearman correlation because we care about getting the ranking of word pairs correct, not necessarily the exact numerical values. If a model assigns similarity 0.82 to "cat"/"dog" and 0.41 to "cat"/"bicycle," and humans rate those pairs as 8.5 and 2.1 on a 10-point scale, the ranking is correct even though the absolute values are on different scales. Spearman correlation measures this rank agreement directly.

A second form of intrinsic evaluation uses word analogy tasks, made famous by Mikolov et al.'s 2013 word2vec paper. An analogy task presents a question of the form "man is to woman as king is to ___?" and asks the model to find the word that completes the analogy. The standard approach is to compute:

and then search for the word in the vocabulary whose vector is nearest to this computed target. If the nearest word is "queen," the analogy is considered solved.

The Google analogy dataset contains 19,544 analogy questions organized into semantic categories (capital cities, country-currency pairs, family relations) and syntactic categories (comparative adjectives, verb tenses, plurals). Word2vec achieves around 65 to 70% accuracy on this dataset. Count-based distributional models typically score lower, around 40 to 55%, motivating the move to neural methods.

Analogy accuracy measures some properties but not others. A model that solves analogies correctly has learned that certain dimensions of the distributional space correspond to systematic semantic transformations (gender, nationality, tense). This is evidence that the distributional representation encodes structured semantic knowledge. But analogy accuracy can be inflated by corpus biases (if the training corpus contains many sentences about capital cities, capital-city analogies will be easy), and it does not directly measure whether the model understands the analogies in a deep sense.

Extrinsic evaluation measures how distributional representations improve performance on downstream NLP tasks when used as features or initialization. Common evaluation tasks include:

• Text classification: Using word vectors as input features and measuring classification accuracy on sentiment analysis, topic classification, or intent detection.
• Named entity recognition: Using word vectors as part of a sequence labeling model and measuring F1 on entity boundaries.
• Parsing: Using word vectors as features in dependency or constituency parsing and measuring labeled attachment scores.
• Question answering: Using word vectors in an end-to-end model and measuring exact match accuracy on reading comprehension benchmarks.

Extrinsic evaluation is more meaningful than intrinsic evaluation for applications: you care whether the representations improve the target task, not whether they correlate with human similarity ratings in the abstract. However, extrinsic evaluation is slower, task-specific, and makes it harder to isolate what property of the representations is responsible for performance differences.

Everything we have demonstrated so far used a small toy corpus of sixteen sentences. A natural question is: how does the quality of distributional representations change as corpus size grows?

The relationship between corpus size and representation quality is approximately logarithmic. Early gains from adding more data are large: going from a hundred sentences to a million substantially improves similarity estimates. But at some point, additional data produces diminishing returns, especially for high-frequency words whose distributions are already well-estimated.

The important caveat is domain. A distributional model trained on legal texts will have excellent representations for legal vocabulary and poor representations for colloquial language. A model trained on social media text will handle informal language well but struggle with technical terms. This domain dependence is not a bug; it is an accurate reflection of the fact that distributional meaning is relative to a community of language users and their texts.

Corpus quality also matters. A clean, well-curated corpus of edited prose often produces better distributional representations than a larger but noisier corpus from the web, for tasks that require precise semantic similarity. This is one reason why early word embedding papers (word2vec, GloVe) were trained on carefully selected corpora like Wikipedia and news archives rather than raw web crawls.

For rare words, corpus size is critical. A word appearing fewer than fifty times in a corpus will have a noisy, unreliable distributional vector. This is one reason why neural embedding methods like word2vec, which can share statistical strength across words through the embedding space, often outperform count-based methods for rare vocabulary.

The distributional hypothesis is powerful, but it has real and important limitations. Understanding them is essential for knowing when to trust distributional methods and when they will mislead you.

Distributional representations are only as good as the corpus they are derived from. A medical corpus will yield excellent similarities among clinical terms but poor similarity estimates for everyday language. A corpus of news articles will encode the semantic associations that appear in news: political figures will be similar to each other, but a child's vocabulary may not be well-represented at all.

This practical engineering limitation also reflects a deeper property: meaning, for distributional methods, is always relative to a particular discourse community and text collection. Words do not have absolute distributional representations. They have corpus-specific ones. The word "mouse" will be dominated by computer-related contexts in a technology corpus and by animal-related contexts in a nature corpus, producing distinct distributional representations that are each locally valid.

The implication is that distributional models trained on general-purpose text may fail on specialized domains, and domain-specific models may not generalize. This is a fundamental tension that has motivated domain adaptation techniques and the creation of large, diverse training corpora.

One persistent failure mode is that distributional similarity conflates semantic relations that humans consider very different. "Hot" and "cold" are antonyms, but they appear in almost identical contexts: weather reports, descriptions of food, discussions of temperature. Their distributional profiles are nearly identical, giving them high cosine similarity. The same problem arises with other near-antonym pairs like "increase" and "decrease," "buy" and "sell," "love" and "hate."

Distributional methods capture the fact that these words appear in semantically related domains. They do not capture the fact that these words express opposed meanings within those domains. This is not a fixable bug in the weighting or similarity measure. It is a consequence of the fact that antonyms are defined by their opposition within a shared semantic field, and distributional evidence cannot distinguish "applies to the same domain" from "means the same thing."

Resolving this requires either richer contextual representations (like the sentence-level representations produced by transformers, which can detect negation and contrastive framing) or explicit negative relation labeling from resources like WordNet. In practice, downstream tasks that require distinguishing synonyms from antonyms often need to combine distributional evidence with other sources of information.

Distributional semantics characterizes individual words well but struggles with phrases. "Not good" and "good" will have similar word vectors because "good" dominates the compositional signal. "White house" and "White House" are distributional neighbors of different things depending on capitalization and context. "Kick the bucket" does not mean anything close to the sum of the vectors for "kick," "the," and "bucket."

The compositionality problem is fundamental: language builds meaning by combining words in structured ways, and a theory that treats each word independently cannot easily capture this. Simple compositional approaches, like averaging word vectors or summing them, produce useful approximate representations for some tasks but fail on idioms, negation, and fine-grained semantic distinctions.

Phrase-level and sentence-level representations, such as those produced by LSTM encoders and transformer models, are partial answers to this problem. These models learn representations that depend on the entire context of a sentence, not just on individual word identities. But even transformer-based representations face challenges with compositional semantics, particularly for novel idioms and complex negation chains.

High-frequency words like "the," "of," "and" co-occur with virtually everything, so their raw co-occurrence vectors are high but uninformative. Rare words have sparse vectors with high statistical uncertainty. A word appearing only five times in the corpus has a co-occurrence profile that may not reflect its true distributional behavior.

Various weighting schemes address the frequency bias. Pointwise Mutual Information (covered in the next chapter) down-weights coincidental co-occurrences caused by overall word frequency. Singular Value Decomposition (covered in a later chapter) smooths sparse vectors by finding a lower-dimensional approximation. Sub-sampling frequent words during training, a technique used in word2vec, directly reduces the influence of stop words on the learned representations.

Addressing data sparsity is harder. For rare words, any statistical estimate of their distribution is unreliable. Neural embedding methods help here because they can generalize across similar words, but the fundamental problem remains: you cannot learn a reliable distributional representation from a handful of examples.

Distributional methods learn from text, and text is not the world. A model trained on text knows that "sun" and "moon" are both discussed in astronomical contexts. It has no perceptual experience of brightness or gravitational pull. "Fire" and "burn" co-occur, but the model has no representation of heat or pain. "Red" is distributionally similar to "blue" and "green," but the model has no representation of wavelengths or visual experience.

This is sometimes called the symbol grounding problem: symbols (words) refer to things in the world, but distributional models contain only relationships between symbols. The meaning of "red" cannot be fully captured by listing the words that tend to appear near it, because part of what "red" means is a particular kind of perceptual experience.

Whether this is a fundamental limitation or a solvable one by training on text at sufficient scale is an active area of debate. Large language models trained on web-scale text can pass verbal descriptions of perceptual tasks (explaining that fire is hot, that red is a warm color) without having any perceptual experience. But whether this constitutes grounded understanding or sophisticated pattern matching remains contested.

Multimodal models, trained on both text and images or text and audio, represent one approach to grounding distributional representations in perceptual experience. These models learn representations that connect verbal and perceptual information, addressing the symbol grounding problem more directly than pure text-based distributional models.

Distributional methods assign a single vector to each word, regardless of how many distinct meanings that word has. "Bank" in "river bank" and "bank" in "financial bank" are averaged together into a single distributional profile. In a large, general corpus, the resulting vector will be somewhere between these two senses, accurately representing neither.

This is the polysemy problem. It becomes acute for words with many distinct senses (highly polysemous words) and for words whose senses appear in very different contexts. A word like "spring" (the season, a coiled object, a source of water, the act of jumping) will have a distributional profile that conflates all four senses.

Sense disambiguation techniques attempt to assign each occurrence of a word to a specific sense before building the distributional representation, but this requires separate sense inventories (like WordNet) and additional computational machinery. Contextual embedding models like ELMo and BERT, which produce different embeddings for the same word in different sentence contexts, address polysemy more naturally by generating use-specific rather than type-specific representations.

The distributional hypothesis did not end with count-based co-occurrence matrices. It became the theoretical justification for the entire lineage of dense word representations.

Word2vec, introduced by Mikolov and colleagues in 2013, operationalized the distributional hypothesis through a prediction objective. Instead of counting co-occurrences and storing them in a sparse matrix, word2vec trained a neural network to predict context words from a target word (skip-gram) or to predict a target word from context words (CBOW). The weights of this network, trained on billions of words, became the dense word embeddings. The distributional hypothesis is implicit in the training signal: the model is rewarded for learning representations that predict context, which means it must encode the distributional profile of each word.

GloVe (Global Vectors for Word Representation), introduced by Pennington and colleagues in 2014, made the connection to distributional matrices explicit. GloVe directly factorizes the log of the co-occurrence matrix, learning dense embeddings that capture the same information as the sparse matrix in a lower-dimensional form. The distributional hypothesis is not an assumption in GloVe; it is the defining feature of the objective function.

Transformer language models, from BERT to GPT, take the distributional hypothesis to its logical conclusion. Instead of learning a single distributional vector per word type, they learn context-dependent representations that encode the distributional properties of a word given its specific sentence context. The "company" a word keeps is now its entire sentence, rather than a local window, and the representation changes accordingly. BERT's contextual embeddings for the two senses of "bank" in different sentences are different vectors, addressing the polysemy problem that plagued earlier distributional methods.

The distributional hypothesis has thus served as both the theoretical foundation and the practical motivation for decades of progress in word representation. Every time we use cosine similarity to compare word embeddings, or train a model to predict context, or evaluate whether two words are semantically similar, we are building on Firth's insight that "you shall know a word by the company it keeps."

The distributional hypothesis states that words appearing in similar contexts have similar meanings. This seemingly simple idea is the theoretical cornerstone of distributional semantics and, by extension, of modern NLP.

Key takeaways from this chapter:

• Firth's principle, "you shall know a word by the company it keeps," provides a computational approach to meaning: measure context, measure similarity. This principle holds because language is a communicative activity, and words that serve the same communicative functions appear in the same kinds of contexts.
• Context windows define "context" operationally. Small windows ($k \leq 3$) capture tight syntactic and local semantic patterns. Larger windows ($k \geq 5$) capture broader topical associations. The window size is a hyperparameter that controls what type of semantic relation the model is sensitive to.
• Syntagmatic relations capture co-occurrence in sequence (bread and butter). Paradigmatic relations capture substitutability in the same syntactic slot (dog and cat). Window size and co-occurrence type influence which relation is more prominent in the distributional representation.
• Cosine similarity is the standard measure for comparing distributional vectors. It normalizes for frequency and compares the shape of contextual profiles, correctly identifying words that appear in the same proportional contexts as similar even when their absolute frequencies differ.
• The co-occurrence matrix is the concrete implementation of the distributional hypothesis: a $|V| \times |V|$ matrix where each row is the distributional profile of one word, measured across the entire vocabulary of context words.
• Distributional methods have real limits: they confound synonymy and antonymy, struggle with compositionality, are corpus-dependent, fail on rare words, assign a single representation to polysemous words, and lack world knowledge (the symbol grounding problem).
• The distributional hypothesis directly motivates word2vec, GloVe, and transformer language models. Dense word embeddings are distributional representations learned through neural prediction objectives rather than explicit counting.

In the next chapter, we'll build on these ideas by constructing full co-occurrence matrices and exploring how they encode word relationships at scale. Then we'll tackle PMI weighting, which corrects for the frequency bias that plagues raw co-occurrence counts, and singular value decomposition, which compresses the sparse co-occurrence matrix into dense, generalizable representations.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the distributional hypothesis.