Co-occurrence Matrices › Symmetric vs. Directional Contexts

Most implementations build symmetric matrices: if "coffee" appears 3 positions to the right of "drank," that counts just as much as "coffee" appearing 3 positions to the left. But some tasks benefit from tracking direction.

In a symmetric context window, the matrix entry $C_{ij}$ counts all occurrences of word $j$ near word $i$, regardless of whether $j$ came before or after $i$. This makes $C$ approximately symmetric: $C_{ij} \approx C_{ji}$. The matrix treats "before" and "after" contexts identically.

In a directional (or asymmetric) context, the matrix tracks whether word $j$ appeared to the left or to the right of word $i$. This doubles the effective vocabulary of context features: instead of one count per word pair, you get two: left-occurrence and right-occurrence. Alternatively, directionality can be used for dependency-based contexts, where context is defined by grammatical relationships rather than positional proximity.

Directional contexts capture linguistic asymmetry. In English, adjectives typically precede nouns ("red apple"), so the left context of "apple" frequently contains adjectives while the right context does not. A symmetric matrix blurs this distinction. A directional matrix preserves it, which can be useful for capturing part-of-speech or syntactic roles.

Consider the word "quickly." To its left, you typically find nouns or clause boundaries. To its right, you typically find nothing relevant (it often comes at the end of a verb phrase). The word "running," by contrast, frequently has a subject (noun) to its left and an object or complement to its right. These asymmetric profiles carry rich syntactic information that symmetric counting discards.

Directional contexts also help with discovering semantic relations that have a direction. "Paris is the capital of France" implies a directed relationship between "Paris" and "France" that is not symmetric. Dependency-based distributional models (Padó and Lapata, 2007; Levy and Goldberg, 2014) use grammatical parse trees to define context, where context words are those in specific syntactic relationships (subject, object, modifier) to the target word rather than mere positional neighbors. This produces more precise semantic representations but requires a dependency parser as a preprocessing step.

Symmetric matrices are simpler to construct and work well for most semantic similarity tasks. Directional matrices are more common in specialized applications involving dependency parsing or syntactic role analysis. For the remainder of this chapter, we focus on the symmetric case, which is the standard choice in most distributional semantics work.