Singular Value Decomposition › Summary

Singular Value Decomposition converts large, sparse co-occurrence matrices into compact, dense word embeddings by finding the most important linear dimensions of the data. The key ideas from this chapter are:

• SVD decomposes any matrix as $\mathbf{M} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^T$, where the left singular vectors $\mathbf{U}$ define word space, the right singular vectors $\mathbf{V}$ define context space, and the singular values $\mathbf{\Sigma}$ measure how much variance each direction captures
• Truncated SVD retains only the top $k$ components, giving the optimal rank-$k$ approximation by the Eckart-Young-Mirsky theorem: no other rank-$k$ matrix is closer to the original under the Frobenius norm
• LSA applies truncated SVD to word-context or term-document matrices to discover latent semantic dimensions, enabling words that never co-occur directly to have high similarity if they share distributional neighborhoods
• The singular value spectrum guides the choice of $k$: retain components before the elbow in the scree plot, and validate with downstream task performance
• Exact SVD is $O(V^3)$ but sparse matrix algorithms and randomized SVD bring this to practical levels; scikit-learn's TruncatedSVD handles large sparse matrices efficiently with the randomized algorithm
• Randomized SVD achieves near-exact results by projecting to a random low-dimensional subspace first, then computing exact SVD of the smaller projected matrix; this reduces computation from $O(k \cdot \text{nnz})$ to a much faster constant-factor improvement
• SVD dimensions are not directly interpretable but reflect real semantic structure; examining top-loading words for each dimension reveals the semantic contrasts it captures
• LSA captures global co-occurrence patterns but cannot handle polysemy, negation, or compositional meaning; these limitations motivated the transition to neural approaches
• Word2Vec implicitly performs matrix factorization on a shifted PMI matrix, unifying the neural and algebraic traditions and showing that the progress from LSA to Word2Vec was a matter of objective and training procedure rather than a categorical shift in methodology