The Distributional Hypothesis › Context Windows

To operationalize the distributional hypothesis, we need a precise definition of "context." The most common approach uses a context window: a region of text surrounding the target word within which we count co-occurring words.

A window of size $k$ centered on target word $w_i$ includes all words within $k$ positions to the left or right:

$$\text{context}(w_i, k) = \{w_{i-k}, \ldots, w_{i-1}, w_{i+1}, \ldots, w_{i+k}\}$$

where:

• $w_i$: the target word at position $i$ in the corpus
• $k$: the half-width of the context window (positions counted from the target)
• $w_{i-k}, \ldots, w_{i-1}$: the $k$ words to the left of the target
• $w_{i+1}, \ldots, w_{i+k}$: the $k$ words to the right of the target

For example, in the sentence "The quick brown fox jumps over the lazy dog," with $k = 2$ and target word "fox":

• Left context: {"quick", "brown"}
• Right context: {"jumps", "over"}

The choice of $k$ matters enormously and captures different kinds of linguistic information. Small windows (1 to 3 words) capture tight syntactic and local semantic patterns. "Fast" and "rapid" tend to appear before the same nouns, and a tight window catches this. Large windows (5 to 10 or more words) capture topical associations. "Doctor" and "hospital" may not appear within 2 words of each other, but they frequently appear in the same broader region of text.

The window size is best understood not as a binary choice but as a dial that controls what kind of semantic relationship the model emphasizes. The right window size depends on what downstream task you care about. For lexical substitution, "synonyms that can replace each other in a sentence," small windows work better. For topic modeling, "words that appear together in discussions of the same subject," larger windows or document-level co-occurrence is more appropriate.

A practical question is whether the context window should cross sentence boundaries. Most implementations do not let windows span beyond a sentence, because linguistic relationships primarily hold within sentences. A word near the end of one sentence is not semantically related to a word near the beginning of the next just because they happen to be nearby in the raw text stream.

This boundary enforcement also prevents spurious associations. If two unrelated sentences happen to be adjacent in a corpus, we don't want their terminal and initial words to be artificially linked. Respecting sentence boundaries keeps the co-occurrence evidence linguistically grounded.

There is one case where cross-sentence context is sometimes permitted: when modeling discourse coherence or long-range topical associations. In these cases, document-level co-occurrence (counting whether two words appear anywhere in the same document, regardless of position) is often more appropriate than a window that straddles sentence endings.

One refinement is to give closer context words more weight than distant ones. A word appearing immediately next to the target is more likely to be semantically relevant than one five positions away. A distance-weighted count for position offset $\delta$ might use a weight of $1/|\delta|$, giving:

$$\text{weighted\_count}(w_i, c, k) = \sum_{\delta = -k,\, \delta \neq 0}^{k} \frac{1}{|\delta|} \cdot \mathbf{1}[w_{i+\delta} = c]$$

where:

• $w_i$: the target word
• $c$: a candidate context word
• $\delta$: the signed offset from the target position
• $\mathbf{1}[w_{i+\delta} = c]$: an indicator that equals 1 if the word at offset $\delta$ is $c$, and 0 otherwise
• $\frac{1}{|\delta|}$: the inverse-distance weight, higher for closer positions

This weighting is used in some early word vector methods and can improve the quality of similarity estimates. Word2vec's skip-gram model, rather than using explicit weighting, effectively achieves a similar result by training on randomly sampled context words, which under uniform sampling produces context distributions concentrated around the target.

The standard context window is symmetric: it extends equally to both sides of the target. But some distributional models use asymmetric windows, considering only left-context or right-context separately. Left and right contexts encode different kinds of information. In English, the left context of a noun often contains adjectives and determiners, while the right context contains verbs and prepositions. Separating these can capture finer-grained syntactic information.

Some models build separate left-context and right-context vectors and concatenate them, giving a richer representation that encodes both what comes before a word and what comes after. This directional approach is particularly useful for tasks that require syntactic sensitivity.