Word Embeddings: Word2Vec, GloVe and Vector Semantics › Limitations & Impact
Word Embeddings: Word2Vec, GloVe and Vector Semantics › Limitations & ImpactOne section · Sections in this article8 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Word embeddings revolutionized NLP, but they have important limitations that contextual embeddings (like BERT) address.
Key Limitations
-
Static representations: Each word has a single embedding regardless of context. "bank" (financial institution) and "bank" (river edge) have the same vector, even though they mean different things.
-
Out-of-vocabulary words: Words not seen during training get no embedding. This is particularly problematic for morphologically rich languages and domain-specific terminology.
-
No sentence-level understanding: Embeddings capture word-level semantics but don't understand how words combine to form meaning. The phrase "not good" might be represented as the average of "not" and "good" embeddings, losing the negation.
-
Bias amplification: Embeddings learn biases present in training data. Gender stereotypes, racial biases, and other problematic associations get encoded in the vector space.
-
Limited to word-level: Can't represent subword units (important for morphologically rich languages) or multi-word expressions without special handling.
Historical Impact
Despite these limitations, word embeddings had significant impact:
- Transfer learning: Pre-trained embeddings enabled NLP models to work with limited task-specific data
- Semantic search: Embeddings power modern search engines and recommendation systems
- Foundation for transformers: The embedding concept evolved into positional encodings and token embeddings in transformers
- Research catalyst: Sparked interest in representation learning that led to contextual embeddings
The Path Forward
Word embeddings were a crucial stepping stone. They proved that dense vector representations could capture semantic relationships, setting the stage for contextual embeddings from transformers. Today, while static word embeddings are rarely used directly in state-of-the-art systems, the principles they established (dense representations, transfer learning, semantic relationships in vector space) remain foundational to modern NLP.