Singular Value Decomposition › Singular Value Decomposition

Co-occurrence matrices and PPMI weighting, as we explored in the previous chapters, give us a way to represent words as high-dimensional vectors where each dimension corresponds to a context word. A word like "bank" might live in a space with tens of thousands of dimensions. These representations capture real associations, but they come with a serious problem: they are enormous and sparse, with substantial noise. Most entries are zero, and the non-zero entries carry substantial statistical noise from the finite size of any real corpus. The question is whether we can do better, whether there is a principled way to compress these unwieldy representations into a compact, dense representation that retains useful information.

Singular Value Decomposition (SVD) is the answer. It is a linear algebra technique that takes a large, sparse matrix and compresses it into a compact set of dense vectors that capture the most important structure while discarding noise. When applied to word-context matrices, this compression produces what are known as Latent Semantic Analysis (LSA) embeddings, an early but remarkably effective form of word embeddings that predates neural approaches by decades. Understanding SVD deeply is worth the effort not just for historical reasons. The mathematical ideas behind SVD, optimal low-rank approximation, orthogonal bases, and the spectral decomposition of data, reappear throughout modern machine learning in forms as varied as principal component analysis, matrix factorization for recommender systems, and even the theoretical analysis of what neural language models are implicitly learning.

The intuition behind SVD is that language has hidden structure. If "dog" and "cat" both appear frequently near "pet," "feed," and "veterinarian," those words are telling us something similar about both animals. But the raw co-occurrence matrix does not know this: it just sees two different rows with different column patterns. SVD discovers these latent patterns by finding directions in the high-dimensional space along which the data varies most strongly. Each direction captures a coherent theme or topic, and projecting words onto the most important directions gives dense, low-dimensional vectors that encode meaning through shared statistical patterns. The dimensions produced by SVD are not labeled with human-interpretable names. They are abstract algebraic constructs, but they encode real semantic relationships that emerge directly from distributional statistics.

This chapter builds up SVD from first principles: why matrix factorization works, what the singular values mean geometrically, how truncation produces optimal embeddings, and how the full pipeline from raw text to cosine similarities operates in practice. We then examine the practical algorithms needed to make SVD feasible at vocabulary scale, and close with an honest assessment of where LSA succeeds and where later approaches were forced to go beyond it.