LSA & Topic Models (1999) › The Solution: Discovering Latent Semantic Structure

Latent Semantic Analysis addressed the vocabulary mismatch problem through a fundamentally different approach: instead of treating words as discrete units, LSA used linear algebra to discover latent semantic dimensions that captured the underlying meaning structures in text collections. The method began by constructing a term-document matrix, where each row represented a unique word in the vocabulary and each column represented a document. The entry at position $(i,j)$ contained the frequency or weighted frequency of word $i$ in document $j$, typically using TF-IDF weighting to emphasize important terms.

The main innovation was applying singular value decomposition (SVD) to this term-document matrix. SVD factorizes any matrix $A$ into three matrices: $A = U \Sigma V^T$, where $U$ contains the left singular vectors, $\Sigma$ is a diagonal matrix of singular values, and $V^T$ contains the right singular vectors. In the context of LSA, this decomposition revealed the latent semantic structure: the columns of $U$ represented semantic dimensions, the singular values in $\Sigma$ indicated the value of each dimension, and the columns of $V^T$ showed how documents were positioned along these dimensions.

LSA then reduced the dimensionality. Instead of using all dimensions, it retained only the $k$ largest singular values and their corresponding vectors, typically choosing $k$ to be much smaller than either the vocabulary size or the number of documents. This low-rank approximation captured the most important semantic patterns while filtering out noise and idiosyncratic word usage. Documents that used different vocabulary but discussed similar topics would end up close together in this reduced-dimensional semantic space.

The mathematical operation that enabled this semantic discovery was the projection of both documents and queries into the reduced semantic space. After computing the SVD and retaining only the top $k$ dimensions, new documents could be mapped into this space by multiplying their term-frequency vectors by the appropriate transformation matrix. Similarly, queries could be projected into the same space, allowing comparison based on semantic similarity rather than exact word matching.

While LSA provided a powerful framework for discovering latent semantic structure, its linear algebra foundation left some questions unanswered. The method didn't explicitly model the probabilistic nature of language, and it was unclear how to interpret the semantic dimensions it discovered. These limitations motivated the development of probabilistic topic models in the late 1990s, which explicitly modeled documents as mixtures of topics and topics as distributions over words.

Probabilistic Latent Semantic Indexing (pLSI), developed by Thomas Hofmann, extended the ideas of LSA by providing a probabilistic generative model. Instead of treating documents and words as deterministic vectors, pLSI modeled the probability that a word would appear in a document as a mixture over latent topics. Each topic was characterized by a probability distribution over the vocabulary, specifying which words were likely to appear when that topic was discussed. Documents were represented as mixtures of these topics, with different documents emphasizing different combinations of topics.

The generative process in pLSI assumed that documents were created by repeatedly sampling topics from the document's topic distribution, then sampling words from the chosen topic's word distribution. This probabilistic framework provided a principled way to understand how latent semantic structure gave rise to observed text. It also enabled more sophisticated inference techniques, using methods like expectation-maximization to learn the topic distributions from observed documents.

The key advantage of the probabilistic approach was its interpretability. Each discovered topic could be understood as a probability distribution over words, making it clear which terms were most characteristic of that topic. A topic about "climate change" might assign high probabilities to words like "temperature," "carbon," "emissions," and "atmosphere," while assigning low probabilities to unrelated terms. Documents could be summarized by their topic mixtures, showing what proportion of the document was devoted to each discovered theme.

The probabilistic framework also enabled principled handling of uncertainty. Rather than making deterministic assignments, pLSI could express degrees of belief about which topics were present in a document and to what extent. This uncertainty modeling proved valuable for applications like information retrieval, where systems needed to make decisions in the face of ambiguous or noisy text.

The process of learning topic models from a document collection involved probabilistic inference: given the observed words in documents, what were the most likely topic distributions? This was typically solved using the expectation-maximization (EM) algorithm, which alternated between two steps. The E-step estimated the probabilities that each word in each document was generated by each topic, given the current estimates of topic distributions. The M-step updated the topic distributions to maximize the likelihood of the observed words under these probabilistic assignments.

This iterative process converged to a solution that discovered coherent topics as collections of words that frequently co-occurred across documents. The algorithm automatically identified that words like "gene," "protein," "DNA," and "mutation" tended to appear together, forming a topic about molecular biology. Similarly, it would discover that "market," "trading," "stock," and "investment" formed a topic about finance, without requiring any explicit labels or supervision.

The beauty of this approach lay in its ability to discover structure that wasn't explicitly encoded in the data. Documents didn't come labeled with topics. The vocabulary didn't include explicit semantic categories. Yet through probabilistic inference over word co-occurrence patterns, the model could discover meaningful thematic structure that captured the underlying semantic organization of the text collection.