Co-occurrence Matrices › Summary

Co-occurrence matrices transform the distributional hypothesis into a concrete data structure. The key ideas are:

• Word-word matrices count how often words appear in each other's context window, capturing local semantic and syntactic relationships. Each row is a distributional signature for a word in terms of its context neighbors.
• Word-document matrices count word occurrences across documents, capturing topical and thematic relationships. They form the basis of TF-IDF and Latent Semantic Analysis.
• Window size controls the type of relationship captured: small windows (1-2) emphasize syntactic patterns; large windows (5-10) emphasize semantic and topical similarity.
• Distance weighting (typically $1/d$) reduces the influence of distant context words, amplifies signal from adjacent words, and improves the quality of downstream word vectors.
• Symmetric vs. directional contexts determine whether the position of context words relative to the target is tracked. Symmetric is standard for semantic similarity; directional contexts are used in dependency-based distributional models.
• Sparsity is unavoidable: most word pairs never co-occur, so sparse matrix formats (COO, CSR, hash maps) are essential for tractable storage and computation.
• Efficient construction streams through the corpus once, building a hash map of (word, context) counts before converting to sparse matrix format. Parallelization allows Wikipedia-scale corpora to be processed in hours.
• Function word dominance and the power law structure of counts motivate statistical reweighting. Raw counts alone are poor semantic representations.

The limitations of raw co-occurrence matrices, particularly the dominance of high-frequency words and the curse of dimensionality, motivated two important extensions: PMI weighting (next chapter) to rescale counts by statistical association, and SVD (a later chapter) to compress the matrix into dense, informative representations. Together, these three techniques form the classical pipeline for distributional word representations, and they remain the conceptual foundation on which every subsequent advance in word embeddings was built.