Singular Value Decomposition › Limitations and Impact

When introduced in 1990, LSA demonstrated that automatic, unsupervised learning could produce word representations capturing semantic similarity, without any hand-built lexicons, dictionaries, or rules. The system outperformed human performance on synonym selection tasks and enabled document retrieval that found relevant documents even when query terms did not appear in them verbatim. These results were remarkable, and they established distributional methods as a serious approach to computational semantics.

The impact extended well beyond information retrieval. LSA became a widely used tool in cognitive science: researchers used it to model how humans acquire vocabulary and how semantic memory is organized. Landauer and Dumais used LSA to argue that word co-occurrence alone, without any symbolic processing, could explain a substantial fraction of human semantic knowledge. This sparked a productive debate about the sufficiency of distributional information for capturing meaning, a debate that continues today in discussions of large language models.

Yet LSA has significant limitations that motivated the development of newer methods. The most fundamental is computational. SVD on large matrices is expensive, and even with randomized algorithms, the memory requirements for large vocabulary matrices are substantial. A production-scale vocabulary of 500,000 words produces a matrix with 250 billion potential entries. Even in sparse form with 0.1% fill, that is 250 million nonzeros at several gigabytes. The full pipeline from corpus to embeddings can take hours on modern hardware for large corpora.

More importantly, LSA operates on global co-occurrence statistics: each entry in the PPMI matrix summarizes co-occurrence across the entire corpus. This means all context is pooled together, losing positional information. The phrase "not good" co-occurs with "good" in a window, but so does "very good." LSA cannot distinguish which pattern is which because the same context word "good" gets counted in both. Negation, scope, and composition are invisible to the global count aggregation.

LSA embeddings also struggle with polysemy. The word "bank" receives a single embedding vector, regardless of whether it appears in a financial context or a geographical one. This single-vector limitation means that LSA conflates the financial sense and the riverbank sense into one mixed representation that may not be ideal for either downstream task. Word2Vec and later dense neural embeddings still share this limitation, but contextual models like ELMo and then BERT eventually solved it by producing different embeddings for each token occurrence depending on the surrounding context.

The reliance on linear algebra places a fundamental ceiling on what LSA can capture. Language meaning is not purely linear: the relationship between "not" and the words it negates requires understanding scope and semantic composition. The meaning of "kick the bucket" (idiomatically: to die) is not a linear combination of the meanings of "kick" and "bucket." Simple matrix factorization cannot capture these nonlinear relationships. Neural embedding methods, starting with Word2Vec in 2013, moved beyond these constraints by learning embeddings through prediction tasks. And transformer-based models moved even further by learning contextual, compositional representations through self-attention over the entire sequence.

Despite these limitations, LSA remains widely used today in specific applications: information retrieval, document similarity, topic exploration, and exploratory analysis of large text collections. Its theoretical guarantees (optimal rank-$k$ approximation by the Eckart-Young theorem), interpretable components, and deterministic behavior (with fixed random seed for randomized variants) make it a principled baseline against which newer methods are compared. Many practitioners find that LSA embeddings are surprisingly competitive with neural embeddings on tasks where global co-occurrence patterns are more informative than local context, particularly in domain-specific retrieval applications.

One of the most intellectually interesting aspects of LSA's legacy is its relationship to Word2Vec. In 2014, Levy and Goldberg proved that Word2Vec's skip-gram model with negative sampling is implicitly factorizing a shifted version of the PMI matrix. The neural network that Word2Vec trains is, in a precise mathematical sense, performing matrix factorization on a modified co-occurrence matrix. This result unified the statistical and neural approaches to distributional semantics, showing that the two superficially different frameworks are mathematically equivalent under certain conditions.

The implication is that the improvements of Word2Vec over LSA are not primarily due to neural networks per se, but rather due to differences in the objective (PMI with negative sampling vs. PPMI), the training procedure (stochastic gradient descent on pairs vs. batch SVD on the full matrix), and the weighting scheme (harmonic weighting of context positions vs. uniform window counts). These are meaningful differences, but they are not the difference between "neural" and "classical": they are differences in how the distributional statistics are collected and how the factorization is performed.

This insight directs attention toward what the actual engineering choices are in building good distributional embeddings, and it clarifies that the progress from LSA to Word2Vec to GloVe was incremental and cumulative, not a revolutionary break. The next chapter covers Word2Vec in detail, which builds directly on the distributional intuitions developed here while introducing the prediction-based training objective that proved empirically superior.

One practical aspect of LSA that practitioners learn quickly is its sensitivity to corpus quality. Unlike neural models, which can tolerate some noise through their implicit regularization mechanisms, SVD is quite sensitive to how the input matrix is constructed. Choices that matter significantly include:

Minimum frequency thresholds: Words that appear fewer than 5-10 times provide unreliable statistics. Including them adds noise to the low singular values and can degrade embedding quality for frequent words through the orthogonality constraint. A common rule of thumb is to exclude words appearing fewer than 5-10 times per million tokens.

Context window size: Narrower windows produce syntactically focused embeddings; wider windows produce topically focused embeddings. The right size depends on the task. Named entity recognition benefits from narrow windows; document similarity benefits from wide ones.

Subsampling: Discarding a fraction of occurrences of very frequent words (analogous to Word2Vec's subsampling) can improve embedding quality by reducing the dominance of the first SVD dimension, which often captures little more than frequency.

PPMI smoothing: Replacing the standard denominator in PPMI with a raised-power version of the context distribution (context probability raised to the power 0.75) reduces the bias toward rare contexts and improves embedding quality on word similarity benchmarks.

These choices interact with $k$ in ways that are hard to predict theoretically, making empirical tuning essential for production systems.