Co-occurrence Matrices › Context Windows

For word-word matrices, the most important construction parameter is the context window. The window defines the neighborhood around each word that counts as "context." Getting this right, or at least understanding the consequences of different choices, is fundamental to building useful co-occurrence representations.

A symmetric window of size $k$ means that for each target word, we count co-occurrences with the $k$ words to its left and $k$ words to its right. Choosing $k$ is not arbitrary: it controls what kind of relationship the matrix captures.

Small windows ($k = 1$ or $2$) capture tight syntactic relationships. The words appearing immediately before or after a target word are often its grammatical neighbors: adjective-noun pairs, verb-object pairs, preposition-noun pairs. A window of 1 for "drank" will frequently find "he," "coffee," or "quickly." These are functional relationships that encode how a word behaves in sentences.

Large windows ($k = 5$ to $10$) capture semantic and topical relationships. Words appearing within a broader span tend to share the same topic or discourse. A window of 10 for "surgery" will capture "doctor," "hospital," "recovery," "anesthesia," and "patient." These are thematic relationships that encode what domain a word inhabits.

This window size effect is well-documented empirically. Levy and Goldberg (2014) showed that small windows produce vectors where nearest neighbors are words in the same syntactic role, while large windows produce vectors where nearest neighbors are topically similar words. Neither is wrong: they answer different questions about meaning.

To develop intuition, consider the word "bank." With a small window of 1-2, "bank" will co-occur with words like "river," "muddy," "steep" (in geographic contexts) and "loan," "branch," "account" (in financial contexts). The ambiguity is preserved because both senses occur, but the specific neighbors reveal which sense is active in each context. With a large window, "bank" in financial contexts will co-occur with a broad cloud of financial vocabulary, making the financial sense more statistically prominent if the corpus is business-heavy.

The window size also interacts with the genre and style of the corpus. In formal written prose, sentences tend to be long and informationally dense, so a window of 5 captures meaningful co-occurrences across clause boundaries. In conversational text or tweets, sentences are short and context is sparser, so even a window of 2 may span most of the sentence.

A practical consideration is whether context windows should cross sentence boundaries. Most implementations do not let context cross sentence boundaries. The sentence "The cat sat on the mat. The dog ran away." should not count "mat" as a context word for "dog," even though "mat" appears within a wide window if you naively count character positions.

Treating the sentence as a natural context unit is linguistically motivated. Sentences are coherent units of meaning, and co-occurrence within a sentence reflects semantic relationships better than co-occurrence across an arbitrary character span. The period between two sentences is a hard semantic break: the ideas being expressed are different, and the words on opposite sides of that break may have nothing to do with each other.

There are cases where crossing paragraph or section boundaries is useful, particularly for long-range coreference or topical coherence tasks. But for standard word-word co-occurrence construction, sentence-bounded windows are the norm.

Some implementations also discard stopwords from context counts. The idea is that "the" and "of" and "in" appear near almost everything and thus carry no discriminative information about word meaning. Including them creates massive counts for common words that dominate the matrix without contributing semantic signal. However, whether to remove stopwords is a design choice. In some tasks, the presence of function words in context is itself informative about grammatical structure, and removing them throws away that signal.

For each target word at position $t$ in a sentence of length $L$, with window size $k$, the context positions are:

$$\text{context}(t) = \{p : \max(0, t-k) \leq p \leq \min(L-1, t+k),\ p \neq t\}$$

For each context position $p$, we identify the word at that position and increment the matrix entry $C_{t\_\text{word},\ p\_\text{word}}$ by the appropriate weight (1 for uniform, $1/|t-p|$ for harmonic). This simple loop, applied to every token in the corpus, is all it takes to build the raw co-occurrence matrix.

The resulting matrix $C$ where $C_{ij}$ is large when word $j$ is a frequent context neighbor of word $i$. The row vector $\mathbf{c}_i$ for word $i$ is its distributional signature: the fingerprint of its verbal environment across the entire corpus.