Co-occurrence Matrices › Two Kinds of Co-occurrence

There are two foundational ways to count co-occurrences, and they answer slightly different questions about language. The distinction is not merely technical. It reflects a fundamental choice about what "context" means: is it the local verbal neighborhood of a word, or the broader documentary environment in which the word appears?

A word-word co-occurrence matrix counts how many times each pair of words appears within a defined proximity in text. If the vocabulary has $V$ words, the matrix has shape $V \times V$. Entry $C_{ij}$ records how often word $j$ appears within the context window of word $i$.

Each row of this matrix is a vector representing a word in terms of its context neighbors. Two words that appear in similar contexts will have similar row vectors, making cosine similarity over these rows a natural measure of semantic relatedness. This is the matrix that directly operationalizes the distributional hypothesis from the previous chapter: "You shall know a word by the company it keeps."

The word-word matrix is the most natural instantiation of local distributional semantics. When linguists studied word meaning in the 1990s and observed that "surgeon" and "physician" had nearly identical row vectors even though they were not perfect synonyms, it was the word-word matrix that demonstrated this computationally for the first time. The vectors do not represent isolated words; they represent words in relation to all other words in the language.

An important property of word-word matrices is that they are approximately symmetric when built with a symmetric window. If "coffee" often appears near "drank," then "drank" often appears near "coffee," because these are the same co-occurrence events observed from different perspectives. The matrix is not perfectly symmetric due to boundary effects at sentence edges, but it is close, and many analyses exploit this symmetry.

A word-document co-occurrence matrix has a different shape: $V \times D$, where $D$ is the number of documents in the corpus. Entry $M_{id}$ records whether (or how many times) word $i$ appears in document $d$.

This matrix encodes a different kind of distributional signal. Instead of asking "which words appear near this word?" it asks "which documents contain this word?" Words that co-occur in the same documents tend to be topically related. "Surgery" and "anesthesia" both appear in medical articles. "Volatility" and "hedge" both appear in financial reports. Document-level co-occurrence captures topic coherence rather than local grammatical or semantic context.

The word-document matrix is also the foundation of information retrieval. When a search engine represents a document as a vector of word counts, it is constructing a column of this matrix. The classic TF-IDF representation, which we examined in Part 1, is simply a reweighted version of this matrix where raw counts are scaled by inverse document frequency. Latent Semantic Analysis (LSA), which we will cover in the SVD chapter, applies Singular Value Decomposition to this matrix to discover latent topics that cut across the raw count structure.

One practical advantage of the word-document matrix over the word-word matrix is that it scales more gracefully when the vocabulary is large but the document collection is modest. A corpus of 10,000 documents with a vocabulary of 100,000 words produces a $100{,}000 \times 10{,}000$ matrix, which is rectangular and often more tractable than the $100{,}000 \times 100{,}000$ word-word matrix.

The choice between word-word and word-document matrices is a modeling decision, and the right choice depends on the relationship you want to capture.

Word-word matrices work better for capturing syntactic relationships and local semantic similarity. If you want to find words that behave grammatically alike, or words that appear in the same sentence-level constructions, the word-word matrix with a small window is the right tool. Words like "run" and "sprint," or "angry" and "furious," tend to show up in the same grammatical positions and thus accumulate similar word-word co-occurrence profiles.

Word-document matrices capture broader thematic associations. If you want to find words that belong to the same topic or domain, the word-document matrix is more appropriate. "Algorithm," "complexity," and "runtime" may not appear adjacent to each other often, but they co-occur in the same computer science articles, and that shared documentary environment marks them as thematically related.

There is a sense in which the word-document matrix is like a very wide word-word matrix. If you set the context window in a word-word matrix to span the entire document, you recover something close to the word-document matrix. The document is the limiting case of context window expansion.