The Distributional Hypothesis › Summary

The distributional hypothesis states that words appearing in similar contexts have similar meanings. This seemingly simple idea is the theoretical cornerstone of distributional semantics and, by extension, of modern NLP.

Key takeaways from this chapter:

• Firth's principle, "you shall know a word by the company it keeps," provides a computational approach to meaning: measure context, measure similarity. This principle holds because language is a communicative activity, and words that serve the same communicative functions appear in the same kinds of contexts.
• Context windows define "context" operationally. Small windows ($k \leq 3$) capture tight syntactic and local semantic patterns. Larger windows ($k \geq 5$) capture broader topical associations. The window size is a hyperparameter that controls what type of semantic relation the model is sensitive to.
• Syntagmatic relations capture co-occurrence in sequence (bread and butter). Paradigmatic relations capture substitutability in the same syntactic slot (dog and cat). Window size and co-occurrence type influence which relation is more prominent in the distributional representation.
• Cosine similarity is the standard measure for comparing distributional vectors. It normalizes for frequency and compares the shape of contextual profiles, correctly identifying words that appear in the same proportional contexts as similar even when their absolute frequencies differ.
• The co-occurrence matrix is the concrete implementation of the distributional hypothesis: a $|V| \times |V|$ matrix where each row is the distributional profile of one word, measured across the entire vocabulary of context words.
• Distributional methods have real limits: they confound synonymy and antonymy, struggle with compositionality, are corpus-dependent, fail on rare words, assign a single representation to polysemous words, and lack world knowledge (the symbol grounding problem).
• The distributional hypothesis directly motivates word2vec, GloVe, and transformer language models. Dense word embeddings are distributional representations learned through neural prediction objectives rather than explicit counting.

In the next chapter, we'll build on these ideas by constructing full co-occurrence matrices and exploring how they encode word relationships at scale. Then we'll tackle PMI weighting, which corrects for the frequency bias that plagues raw co-occurrence counts, and singular value decomposition, which compresses the sparse co-occurrence matrix into dense, generalizable representations.