The Distributional Hypothesis › Distributional Semantics in Practice: Scale Effects

Everything we have demonstrated so far used a small toy corpus of sixteen sentences. A natural question is: how does the quality of distributional representations change as corpus size grows?

The relationship between corpus size and representation quality is approximately logarithmic. Early gains from adding more data are large: going from a hundred sentences to a million substantially improves similarity estimates. But at some point, additional data produces diminishing returns, especially for high-frequency words whose distributions are already well-estimated.

The important caveat is domain. A distributional model trained on legal texts will have excellent representations for legal vocabulary and poor representations for colloquial language. A model trained on social media text will handle informal language well but struggle with technical terms. This domain dependence is not a bug; it is an accurate reflection of the fact that distributional meaning is relative to a community of language users and their texts.

Corpus quality also matters. A clean, well-curated corpus of edited prose often produces better distributional representations than a larger but noisier corpus from the web, for tasks that require precise semantic similarity. This is one reason why early word embedding papers (word2vec, GloVe) were trained on carefully selected corpora like Wikipedia and news archives rather than raw web crawls.

For rare words, corpus size is critical. A word appearing fewer than fifty times in a corpus will have a noisy, unreliable distributional vector. This is one reason why neural embedding methods like word2vec, which can share statistical strength across words through the embedding space, often outperform count-based methods for rare vocabulary.