Word Embeddings: Word2Vec, GloVe and Vector Semantics › The Problem with Traditional Representations

Michael BrenndoerferAugust 30, 20252 min read
Word Embeddings: Word2Vec, GloVe and Vector Semantics › The Problem with Traditional RepresentationsOne section · Sections in this article2 / 10 ↓
Sections in this article10
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before word embeddings, NLP systems relied on sparse, high-dimensional representations that couldn't capture semantic relationships. Understanding these limitations helps explain why embeddings represented a significant advance in NLP.

One-Hot Encoding: The Baseline

One-hot encoding represents each word as a binary vector where exactly one position is 1 and all others are 0. If your vocabulary has 10,000 words, each word gets a 10,000-dimensional vector:

In[3]:
Code
vocabulary = ["cat", "dog", "bird", "airplane", "car"]
word_to_index = {word: i for i, word in enumerate(vocabulary)}


def one_hot_encode(word, vocab_size):
    vector = [0] * vocab_size
    vector[word_to_index[word]] = 1
    return vector


cat_vector = one_hot_encode("cat", len(vocabulary))
dog_vector = one_hot_encode("dog", len(vocabulary))
airplane_vector = one_hot_encode("airplane", len(vocabulary))
Out[4]:
Console
cat: [1, 0, 0, 0, 0]
dog: [0, 1, 0, 0, 0]
airplane: [0, 0, 0, 1, 0]

Notice that "cat" and "dog" are as different as "cat" and "airplane" in this representation. The dot product between any two different one-hot vectors is always zero, meaning we can't measure similarity. This is a fundamental limitation: one-hot encoding treats all words as equally different.

The Curse of Dimensionality

One-hot encoding also suffers from the curse of dimensionality. With a vocabulary of 50,000 words (typical for English), each word requires a 50,000-dimensional vector. Most of these dimensions are wasted: 99.998% of the vector is zeros. This creates several problems:

  • Storage inefficiency: Storing millions of sparse vectors wastes memory
  • Computational overhead: Operations on sparse vectors are slower
  • No generalization: A model can't learn that "running" and "runs" are related if they're represented as completely orthogonal vectors

The Need for Dense Representations

What we need is a dense representation where:

  1. Similarity is measurable: Words with related meanings have similar vectors
  2. Dimensionality is manageable: Compact vectors (100-300 dimensions) instead of thousands
  3. Relationships are preserved: Analogies like "king - man + woman = queen" emerge naturally

Word embeddings provide exactly this. They learn dense vectors where semantic relationships are encoded in the geometry of the vector space.

Word Embedding

A word embedding is a dense, low-dimensional vector representation of a word learned from text data. Unlike one-hot encoding, embeddings capture semantic relationships: words with similar meanings have similar vectors, and relationships like analogies can be expressed through vector arithmetic.