GloVe › Limitations and Impact

GloVe's impact on NLP was substantial. When the paper appeared in 2014, it provided the first rigorous theoretical connection between count-based and prediction-based methods, helping unify two previously separate research traditions. Before GloVe, it was unclear whether the quality difference between Word2Vec and LSA-style matrix factorization was due to the model architecture, the training objective, the use of global versus local statistics, or some combination. GloVe's derivation showed that the key ingredient was encoding co-occurrence ratios rather than raw probabilities, and that prediction-based methods were implicitly doing something very similar. This theoretical clarity guided subsequent research.

Practically, pretrained GloVe vectors became (and remain) a standard baseline for many NLP tasks. Loading 50 to 300 dimensional GloVe vectors as input features improves performance on tasks like named entity recognition, sentiment analysis, text classification, and question answering without any task-specific embedding training. The vectors transfer well because they capture general English semantics from massive corpora, encoding both syntactic patterns (singular/plural, tense) and semantic relationships (synonyms, antonyms, analogies).

The impact was also pedagogical. GloVe's clean derivation from first principles made it possible to explain exactly why word embeddings work, not just demonstrate that they do. Every step of the derivation, from the ratio insight to the exponential functional form to the bias terms, follows logically. This clarity made GloVe a popular pedagogical tool in NLP courses and textbooks.

GloVe shares fundamental limitations with all static embedding methods. Each word gets exactly one vector regardless of context, so "bank" (financial institution) and "bank" (river bank) map to the same point in embedding space. GloVe cannot distinguish between these two senses. As we will explore in later chapters on contextual representations (ELMo, BERT, and their successors), this limitation becomes critical for language understanding tasks where meaning is highly context-dependent. A polysemous word like "bat" needs a different representation in "the bat hung from the ceiling" versus "she swung the bat", and a single static vector cannot capture this.

GloVe's reliance on pre-computed co-occurrence statistics also means that training cannot incorporate new data incrementally. Adding a new document to the corpus requires recomputing the affected rows of the co-occurrence matrix and retraining. Word2Vec's streaming nature is more amenable to online updates, which matters for applications where the corpus evolves over time, such as social media or news data.

The $V \times V$ co-occurrence matrix poses scalability challenges. For a vocabulary of one million words, even a sparse matrix representation can be very large. In practice, aggressive vocabulary pruning (discarding low-frequency words) is necessary. This means GloVe handles rare words poorly: words seen fewer than the minimum count threshold receive no embedding at all. FastText, covered in the next chapter, addresses this by representing words as bags of character n-grams, allowing it to construct vectors for unseen or rare words by composing their subword representations.

Finally, like Word2Vec, GloVe embeddings encode statistical biases present in the training corpus. Gender stereotypes, cultural biases, and historical associations are all faithfully reproduced in the embedding geometry. The parallelogram model for analogies can reveal these biases: "man is to programmer as woman is to homemaker" is a documented failure mode of embeddings trained on unfiltered web text. Research into debiasing word embeddings (projecting out gender subspaces, adjusting co-occurrence statistics) became an active area precisely because GloVe's theoretical transparency made the bias encoding mechanism easy to analyze. The same geometric structure that makes analogies work also makes biases geometrically accessible and measurable.

A subtler limitation is that GloVe treats all co-occurrence pairs independently. Two words that never co-occur directly but frequently share the same context (for example, "cat" and "feline" might rarely appear in the same sentence but both appear near "meow", "purr", and "fur") will have their similarity captured only transitively through their shared context partners. In practice this works well, but it means the model cannot explicitly represent transitive or higher-order relationships.