The Distributional Hypothesis › Limitations of Distributional Semantics

The distributional hypothesis is powerful, but it has real and important limitations. Understanding them is essential for knowing when to trust distributional methods and when they will mislead you.

Distributional representations are only as good as the corpus they are derived from. A medical corpus will yield excellent similarities among clinical terms but poor similarity estimates for everyday language. A corpus of news articles will encode the semantic associations that appear in news: political figures will be similar to each other, but a child's vocabulary may not be well-represented at all.

This practical engineering limitation also reflects a deeper property: meaning, for distributional methods, is always relative to a particular discourse community and text collection. Words do not have absolute distributional representations. They have corpus-specific ones. The word "mouse" will be dominated by computer-related contexts in a technology corpus and by animal-related contexts in a nature corpus, producing distinct distributional representations that are each locally valid.

The implication is that distributional models trained on general-purpose text may fail on specialized domains, and domain-specific models may not generalize. This is a fundamental tension that has motivated domain adaptation techniques and the creation of large, diverse training corpora.

One persistent failure mode is that distributional similarity conflates semantic relations that humans consider very different. "Hot" and "cold" are antonyms, but they appear in almost identical contexts: weather reports, descriptions of food, discussions of temperature. Their distributional profiles are nearly identical, giving them high cosine similarity. The same problem arises with other near-antonym pairs like "increase" and "decrease," "buy" and "sell," "love" and "hate."

Distributional methods capture the fact that these words appear in semantically related domains. They do not capture the fact that these words express opposed meanings within those domains. This is not a fixable bug in the weighting or similarity measure. It is a consequence of the fact that antonyms are defined by their opposition within a shared semantic field, and distributional evidence cannot distinguish "applies to the same domain" from "means the same thing."

Resolving this requires either richer contextual representations (like the sentence-level representations produced by transformers, which can detect negation and contrastive framing) or explicit negative relation labeling from resources like WordNet. In practice, downstream tasks that require distinguishing synonyms from antonyms often need to combine distributional evidence with other sources of information.

Distributional semantics characterizes individual words well but struggles with phrases. "Not good" and "good" will have similar word vectors because "good" dominates the compositional signal. "White house" and "White House" are distributional neighbors of different things depending on capitalization and context. "Kick the bucket" does not mean anything close to the sum of the vectors for "kick," "the," and "bucket."

The compositionality problem is fundamental: language builds meaning by combining words in structured ways, and a theory that treats each word independently cannot easily capture this. Simple compositional approaches, like averaging word vectors or summing them, produce useful approximate representations for some tasks but fail on idioms, negation, and fine-grained semantic distinctions.

Phrase-level and sentence-level representations, such as those produced by LSTM encoders and transformer models, are partial answers to this problem. These models learn representations that depend on the entire context of a sentence, not just on individual word identities. But even transformer-based representations face challenges with compositional semantics, particularly for novel idioms and complex negation chains.

High-frequency words like "the," "of," "and" co-occur with virtually everything, so their raw co-occurrence vectors are high but uninformative. Rare words have sparse vectors with high statistical uncertainty. A word appearing only five times in the corpus has a co-occurrence profile that may not reflect its true distributional behavior.

Various weighting schemes address the frequency bias. Pointwise Mutual Information (covered in the next chapter) down-weights coincidental co-occurrences caused by overall word frequency. Singular Value Decomposition (covered in a later chapter) smooths sparse vectors by finding a lower-dimensional approximation. Sub-sampling frequent words during training, a technique used in word2vec, directly reduces the influence of stop words on the learned representations.

Addressing data sparsity is harder. For rare words, any statistical estimate of their distribution is unreliable. Neural embedding methods help here because they can generalize across similar words, but the fundamental problem remains: you cannot learn a reliable distributional representation from a handful of examples.

Distributional methods learn from text, and text is not the world. A model trained on text knows that "sun" and "moon" are both discussed in astronomical contexts. It has no perceptual experience of brightness or gravitational pull. "Fire" and "burn" co-occur, but the model has no representation of heat or pain. "Red" is distributionally similar to "blue" and "green," but the model has no representation of wavelengths or visual experience.

This is sometimes called the symbol grounding problem: symbols (words) refer to things in the world, but distributional models contain only relationships between symbols. The meaning of "red" cannot be fully captured by listing the words that tend to appear near it, because part of what "red" means is a particular kind of perceptual experience.

Whether this is a fundamental limitation or a solvable one by training on text at sufficient scale is an active area of debate. Large language models trained on web-scale text can pass verbal descriptions of perceptual tasks (explaining that fire is hot, that red is a warm color) without having any perceptual experience. But whether this constitutes grounded understanding or sophisticated pattern matching remains contested.

Multimodal models, trained on both text and images or text and audio, represent one approach to grounding distributional representations in perceptual experience. These models learn representations that connect verbal and perceptual information, addressing the symbol grounding problem more directly than pure text-based distributional models.

Distributional methods assign a single vector to each word, regardless of how many distinct meanings that word has. "Bank" in "river bank" and "bank" in "financial bank" are averaged together into a single distributional profile. In a large, general corpus, the resulting vector will be somewhere between these two senses, accurately representing neither.

This is the polysemy problem. It becomes acute for words with many distinct senses (highly polysemous words) and for words whose senses appear in very different contexts. A word like "spring" (the season, a coiled object, a source of water, the act of jumping) will have a distributional profile that conflates all four senses.

Sense disambiguation techniques attempt to assign each occurrence of a word to a specific sense before building the distributional representation, but this requires separate sense inventories (like WordNet) and additional computational machinery. Contextual embedding models like ELMo and BERT, which produce different embeddings for the same word in different sentence contexts, address polysemy more naturally by generating use-specific rather than type-specific representations.