The Distributional Hypothesis › Paradigmatic vs. Syntagmatic Relations

One of the most important distinctions in distributional semantics is between two types of word relationships captured by context.

Syntagmatic relations hold between words that commonly appear together, in sequence. "Bread" and "butter," "coffee" and "hot," "kick" and "ball," are syntagmatically related. These words tend to co-occur in the same context windows. If you build a distributional representation that counts direct co-occurrence, syntagmatic relations will be prominent.

Paradigmatic relations hold between words that appear in the same positions relative to other words, but not necessarily together. "Dog" and "cat" are paradigmatically related: both can follow "the," both can be the subject of "ran" or "sat," both appear near "pet" and "feed." These words occupy the same functional slot in sentences. They are substitutable for each other.

Whether a distributional model captures syntagmatic or paradigmatic relations depends heavily on the context window size and the type of co-occurrence counted.

• Small windows and direct co-occurrence counts emphasize syntagmatic relations. Words that appear next to each other will have high overlap.
• Larger windows and especially document-level co-occurrence emphasize paradigmatic (topical) relations. "King" and "queen" both appear in documents about royalty, so their document-level distributions overlap.

This distinction matters practically because different NLP tasks call for different types of similarity. Thesaurus construction requires paradigmatic similarity: you want to find words that mean the same thing, which means they must be substitutable in the same positions. Information retrieval often benefits from syntagmatic similarity: knowing that "search" and "query" appear near similar words helps match a user query to relevant documents.

Word2vec's skip-gram model, which we'll explore in later chapters, was specifically designed with small windows to capture paradigmatic similarity. The classic result that "king - man + woman $\approx$ queen" works because the model learned paradigmatic functional relationships. "King" and "queen" appear in similar syntactic positions (subject of sentences about royalty, object of "crowned"), which means their distributional vectors are similar, and the arithmetic operation captures the gender-role component of that similarity.

Related to the paradigmatic/syntagmatic distinction is the difference between thematic and taxonomic similarity. Thematic similarity groups words that appear together in the same event or scene: "coffee," "mug," "hot," and "morning" are thematically related because they often co-occur in descriptions of the same activity. Taxonomic similarity groups words of the same type: "coffee," "tea," "juice," and "water" are all beverages.

Distributional methods tend to capture both, but the balance shifts with window size and context definition. Tight windows in rich syntactic contexts often capture taxonomic similarity better, because words of the same syntactic category (nouns, verbs, adjectives) appear in structurally similar positions. Looser, topic-oriented windows capture thematic similarity better.

Human semantic memory appears to contain both types of relations, organized in overlapping networks. The distributional hypothesis provides a single framework that, with the right settings, can extract either type from raw text.