The Distributional Hypothesis › The Distributional Hypothesis

What does a word mean? A philosopher might say a word refers to a concept in the mind. A logician might point to truth conditions, the set of objects in the world to which the word correctly applies. But a linguist named John Rupert Firth had a different answer, one that would eventually reshape how computers process language.

Firth's famous observation, stated in 1957, was simple: "You shall know a word by the company it keeps." The idea is that meaning is not some abstract property a word carries around by itself. Meaning emerges from patterns of use. Words that appear in the same kinds of sentences, surrounded by the same kinds of neighbors, tend to mean similar things. This is the distributional hypothesis, and it is the theoretical foundation for nearly everything in modern NLP, from bag-of-words models to word2vec to transformer language models.

This chapter unpacks what the distributional hypothesis says, why it holds, and what it enables. We'll look at context windows, the distinction between paradigmatic and syntagmatic word relations, and how to compute word similarity directly from distributional evidence. We'll also examine its limitations directly, because understanding where a theory breaks down is just as important as understanding where it works.