The Distributional Hypothesis › From Distributional Hypothesis to Neural Embeddings

The distributional hypothesis did not end with count-based co-occurrence matrices. It became the theoretical justification for the entire lineage of dense word representations.

Word2vec, introduced by Mikolov and colleagues in 2013, operationalized the distributional hypothesis through a prediction objective. Instead of counting co-occurrences and storing them in a sparse matrix, word2vec trained a neural network to predict context words from a target word (skip-gram) or to predict a target word from context words (CBOW). The weights of this network, trained on billions of words, became the dense word embeddings. The distributional hypothesis is implicit in the training signal: the model is rewarded for learning representations that predict context, which means it must encode the distributional profile of each word.

GloVe (Global Vectors for Word Representation), introduced by Pennington and colleagues in 2014, made the connection to distributional matrices explicit. GloVe directly factorizes the log of the co-occurrence matrix, learning dense embeddings that capture the same information as the sparse matrix in a lower-dimensional form. The distributional hypothesis is not an assumption in GloVe; it is the defining feature of the objective function.

Transformer language models, from BERT to GPT, take the distributional hypothesis to its logical conclusion. Instead of learning a single distributional vector per word type, they learn context-dependent representations that encode the distributional properties of a word given its specific sentence context. The "company" a word keeps is now its entire sentence, rather than a local window, and the representation changes accordingly. BERT's contextual embeddings for the two senses of "bank" in different sentences are different vectors, addressing the polysemy problem that plagued earlier distributional methods.

The distributional hypothesis has thus served as both the theoretical foundation and the practical motivation for decades of progress in word representation. Every time we use cosine similarity to compare word embeddings, or train a model to predict context, or evaluate whether two words are semantically similar, we are building on Firth's insight that "you shall know a word by the company it keeps."