Singular Value Decomposition › Latent Semantic Analysis

LSA, introduced by Deerwester, Dumais, Furnas, Landauer, and Harshman in their landmark 1990 paper "Indexing by latent semantic analysis," applies truncated SVD to term-document or term-term co-occurrence matrices to obtain word representations. The paper proposed LSA primarily as a method for document retrieval, but its implications for semantic representation quickly became clear.

The name "latent semantic" captures the core idea: the dimensions discovered by SVD correspond to underlying (latent) semantic themes that are not visible in the raw counts. The original LSA paper demonstrated something remarkable: after SVD compression, words that never appear in the same document can still have high cosine similarity in the compressed space, if they tend to appear in documents covering similar topics. This is latent semantics: meaning inferred from indirect co-occurrence patterns, not direct co-occurrence.

For example, "physician" and "doctor" rarely appear in the same sentence. In a raw co-occurrence matrix, their co-occurrence count might be close to zero. But both frequently appear in texts about hospitals, patients, treatments, and medical care. SVD discovers that these words belong to the same semantic neighborhood by noticing that they appear with similar context words across the corpus. The shared context lifts them into the same neighborhood in the latent space, even though they were never direct co-occurrences.

The original LSA evaluation was striking. On synonym selection tests from the Test of English as a Foreign Language (TOEFL), an LSA system trained on a large text corpus achieved accuracy comparable to college-bound non-native English speakers, and substantially above chance. This was achieved without any explicit synonym dictionaries or semantic rules, purely from distributional patterns in raw text. It was early evidence that statistical structure in large corpora encodes something meaningful about semantic relationships.

LSA can be applied to either of two matrix types, and the choice matters for what kind of similarity the resulting embeddings capture.

Term-document matrices have rows for terms and columns for documents. Entry $(i, j)$ records how often term $i$ appears in document $j$, typically with TF-IDF weighting to reduce the dominance of common words. SVD on this matrix discovers patterns of which terms tend to appear together in the same documents. Two words will be similar if they tend to appear in the same kinds of documents, even if they rarely appear in the same sentence. This makes term-document LSA natural for topical similarity: "automobile" and "vehicle" are similar because they both appear in transportation articles.

Term-term co-occurrence matrices (also called word-word matrices) have rows and columns both representing vocabulary words, with entries recording how often the row word appears within a context window of the column word. The window might be 2 words to the left and right, or 5 words, or an entire document. SVD on this matrix discovers patterns of which words appear in similar local contexts. Two words will be similar if they are interchangeable in sentences, not just in documents.

Both approaches produce useful word embeddings, but they capture different aspects of similarity. Term-document matrices tend to capture topical or associative similarity: "surgeon" and "hospital" are similar because they appear in medical documents. Term-term matrices capture paradigmatic or substitutional similarity: "surgeon" and "physician" are similar because they can substitute for each other in sentences ("the surgeon/physician examined the patient"). For most NLP applications, paradigmatic similarity is more useful, which is why term-term matrices have become the standard for distributional word embeddings.

The window size in term-term matrices also matters. Narrow windows (1-2 words) capture syntactic similarity: words that fill the same grammatical role. Wide windows (5-10 words or entire documents) blend into topical similarity. This is because nearby words are constrained by local syntax, while distant words are constrained by document topic. The transition from narrow to wide windows interpolates between syntactic and topical similarity, and the right choice depends on the downstream task.

LSA is grounded in the distributional hypothesis, which holds that words with similar meanings tend to appear in similar contexts. We discussed this hypothesis in the chapter on distributional semantics, and LSA operationalizes it with mathematical precision. The PPMI matrix measures how much each word-context pair co-occurs beyond chance. SVD then finds the directions in this space along which co-occurrence patterns are most coherent and persistent.

What makes LSA's formulation powerful is that it goes beyond pairwise co-occurrence. Two words can be similar in the LSA space even if their direct co-occurrence count is zero, as long as they share indirect neighbors. If "cat" often appears near "pet" and "food," and "dog" often appears near "pet" and "food," then SVD places them near each other because they project strongly onto the same first few left singular vectors. The singular vectors aggregate co-occurrence signals across the entire vocabulary, finding the directions that simultaneously explain many words at once.