The Distributional Hypothesis › The Co-occurrence Matrix as a Word Representation

It is worth stepping back and thinking about what we have built. A co-occurrence matrix is a $|V| \times |V|$ matrix where $|V|$ is the vocabulary size. The entry at row $i$, column $j$ records how often word $i$ appears in the context of word $j$.

where $\text{count}(w_i, w_j, k)$ is the number of times word $w_j$ appears within a window of size $k$ around word $w_i$ across the entire corpus.

This matrix is the distributional representation of the entire vocabulary. Each row is the distributional vector for one word. The columns represent context dimensions. The whole matrix is the computational realization of the distributional hypothesis.

This representation has some striking properties:

• Symmetry: If we use a symmetric context window, $M_{ij} = M_{ji}$ because every time word $j$ appears in the context of word $i$, word $i$ also appears in the context of word $j$. The matrix is symmetric.
• Sparsity: In any real vocabulary, most word pairs never co-occur. A vocabulary of 100,000 words would require a $10^{10}$-entry matrix, the vast majority of whose entries are zero. This sparsity is one motivation for the dimensionality reduction techniques covered in later chapters.
• High dimensionality: Even for our toy corpus, the matrix has thousands of entries. For real corpora, the vocabulary might have hundreds of thousands of words, making the full matrix computationally expensive to store and compare.
• Frequency bias: The matrix directly reflects word frequencies. Common words like "the" have large counts everywhere, distorting similarity estimates toward high-frequency vocabulary.

The progression from raw co-occurrence matrices to PMI-weighted matrices to dense word embeddings (word2vec, GloVe) is the story of how the field addressed these limitations while preserving the core distributional insight.