GloVe & Adam Optimizer (2014) › What They Couldn't Solve

Despite its improvements over word2vec in some areas, GloVe shared the same fundamental limitation: static embeddings. Each word got exactly one vector, regardless of how it was used. The word "bank" had the same representation whether it appeared in "I deposited money at the bank" or "We sat on the river bank." The embedding averaged across all possible meanings, unable to distinguish between financial institutions and geographical features.

This polysemy problem.words having multiple meanings.was inherent to the static embedding approach, not something GloVe could fix by using global statistics. No matter how cleverly you analyzed co-occurrence patterns, if you had to represent each word with a single fixed vector, you couldn't capture context-dependent meanings.

GloVe also faced practical constraints. Building the co-occurrence matrix required substantial memory.for a 50,000-word vocabulary, that's 2.5 billion entries to store and process. While the matrix was sparse (most word pairs never co-occurred), handling truly massive vocabularies or enormous corpora pushed memory limits. Word2vec's streaming approach, which processed text in small batches without building a global matrix, scaled more easily to very large datasets.

Like word2vec, GloVe couldn't handle out-of-vocabulary words. Encounter a word that wasn't in the training data? You had no way to generate a meaningful embedding for it. This was especially problematic for morphologically rich languages (like German or Finnish, where words take many different forms), technical domains (with specialized terminology), and social media (with creative spellings and emerging slang).

The weighting function, while necessary for good performance, added hyperparameters that needed tuning. The defaults ($x_{max} = 100$, $\alpha = 0.75$) worked well, but optimal values varied across different corpora and tasks. And unlike word2vec, which could update incrementally as new text arrived, GloVe required rebuilding the entire co-occurrence matrix when you wanted to incorporate new data. This made it less suitable for applications where the vocabulary or corpus evolved over time.

Adam wasn't perfect. As researchers gained experience with it, they discovered some limitations. In certain situations, particularly with very noisy or sparse gradients, Adam's adaptive learning rates could decrease too aggressively. The optimizer might converge to a solution quickly, but that solution might not be quite as good as what carefully tuned SGD with a learning rate schedule could achieve. This generalization gap.where Adam trained faster but SGD sometimes reached slightly better final performance.became particularly noticeable in some computer vision tasks.

The memory overhead was also non-trivial. Adam stored two additional values per parameter (the first and second moment estimates), effectively tripling the memory needed for optimizer state compared to simple SGD. For massive models with billions of parameters, this memory cost added up. On a model with 1 billion parameters, Adam needed an extra 8 GB of memory just for its internal bookkeeping.

Some researchers discovered that Adam's treatment of weight decay.a regularization technique that prevents parameters from growing too large.wasn't quite right. This led to variants like AdamW that separated weight decay from the adaptive learning rate mechanism, achieving better generalization on some tasks.

There were also subtle numerical issues. The bias correction terms ($1 - \beta_1^t$ and $1 - \beta_2^t$) could become very large in early training steps when $t$ was small, potentially causing numerical instability if not implemented carefully.

Despite these limitations, Adam remained the default choice for most applications. The generalization gap, when it existed, was often small enough that Adam's faster convergence and ease of use outweighed the slight performance difference. And for the vast majority of tasks, Adam's default hyperparameters worked well enough that tuning them provided minimal benefit.