The Distributional Hypothesis › Meaning from Context

Before formalizing anything, let's build intuition. Suppose you encounter the word "wug" in these sentences:

• "The wug ran across the field."
• "She fed the wug a carrot."
• "The wug's fur was soft and brown."
• "A wug can run up to 20 miles per hour."

You've never seen the word before, but you already know quite a lot about it. A wug is an animal. It is probably medium-sized and furry, probably also fast. It eats carrots, which hints at an herbivore. From pure context, with zero dictionary access, you've inferred a rich meaning.

This is what the distributional hypothesis says is happening all the time, with every word, in every language. When a child learns that "dog" and "cat" are both things you feed and pet and that run and play, they are building a distributional profile. When a reader encounters a technical term they've never defined, they infer meaning from the sentences it lives in. When a language model predicts what comes next, it is, in a very real sense, using context to represent meaning.

The distributional hypothesis does not claim that context defines meaning completely. Rather, it makes a weaker and more useful claim: context provides reliable evidence about meaning. Two words with highly overlapping contexts are almost certainly semantically related. This gives us a computational toehold. If we can measure context, we can measure similarity, and similarity is an enormous part of what meaning involves.

The hypothesis also has a deep empirical grounding. Across many languages and many corpora, researchers have confirmed that distributional similarity correlates strongly with human judgments of semantic relatedness. When you ask people to rate how similar "cat" and "dog" are versus "cat" and "bicycle," the distributional estimates track those human intuitions remarkably well. This is not a given. Many things that are logically elegant fail to match empirical data. The fact that distributional profiles correlate with human meaning judgments is what transformed an interesting linguistic observation into a practical engineering tool.

It's worth pausing to ask why this principle holds at all. Why would the contexts a word appears in be informative about its meaning?

A practical answer is that language is a social, communicative activity. Speakers choose words based on what they're trying to convey, and they place those words in sentences that make their meaning recoverable for a listener or reader. If two words can be used in the same communicative contexts, it is because they can serve the same communicative purposes, and that functional equivalence is deeply related to semantic equivalence.

There is also a recursive dynamic at work. The meaning of "dog" is partly determined by the contexts it appears in, but those contexts are themselves composed of other words whose meanings are similarly determined. "Dog" appears near "bark," "leash," "collar," and "fetch," and those words in turn appear near other semantically coherent clusters. The entire lexicon forms a web of mutual constraints, and the distributional hypothesis says we can discover this web by counting co-occurrences.

A third perspective comes from cognitive science. Psycholinguistic studies have shown that when people read sentences, they activate the meanings of words they haven't seen yet based purely on prior context. When you read "She reached into the refrigerator and pulled out a cold..." your brain has already predicted that the next word is likely a food item, probably something you'd store cold. Your mental model of what words can appear in what positions predicts meaning. The distributional hypothesis is essentially a corpus-level formalization of this prediction process.

Zellig Harris, a structural linguist writing in the 1950s, articulated the formal version of this principle first. Harris observed that morphological and syntactic regularities could be discovered by examining distributions of phonemes and words without reference to meaning at all. He called this the "distributional method," and it was meant as a technique for linguistic analysis, not as a theory of semantics.

Firth, working in a different tradition rooted in British contextualism, was more interested in meaning than in structure. His contribution was to take Harris's distributional insight and redirect it toward semantics. His claim went further than treating distributional patterns as useful evidence for linguistic analysis: he argued that they constitute meaning. Words acquire their meanings through the habitual company they keep. A word like "bank" means different things partly because of its different distributional contexts: financial sentences versus river sentences.

These ideas sat largely dormant in computational form until the 1980s and 1990s, when the availability of large digital corpora made it practical to compute distributions at scale. The Brown Corpus, released in 1961, was among the first systematic collections of machine-readable text. By the 1980s, researchers were building gigabyte-scale corpora from newspaper archives. Researchers including Church and Hanks, Turney, and Schütze began developing mathematical frameworks for measuring distributional similarity, laying the groundwork for what we now call word vectors.

Church and Hanks's 1990 paper on Pointwise Mutual Information (PMI) was particularly influential. They showed that co-occurrence counts, properly normalized, could capture specific lexical associations rather than just frequency artifacts. Schütze's work on context vectors in the early 1990s demonstrated that high-dimensional co-occurrence vectors could be compressed into low-dimensional representations that preserved semantic structure. These ideas directly anticipate word2vec, GloVe, and ultimately the attention-based context representations of transformers.