Singular Value Decomposition › Interpreting SVD Dimensions

One of the persistent challenges with SVD-based embeddings is that the resulting dimensions are not directly interpretable. Unlike topic models such as Latent Dirichlet Allocation, which produce explicit probability distributions over words for each topic, SVD dimensions are abstract linear combinations of all the original features. A dimension is defined by its loading vector: the left singular vector $\mathbf{u}_i$, which assigns a real-valued weight to every word in the vocabulary. Positive weights indicate words that are "on one side" of the dimension; negative weights indicate words on the other side.

Despite this abstractness, considerable structure is recoverable through careful inspection. For each SVD dimension, you can rank the vocabulary words by their loading in the corresponding left singular vector: the words with the highest positive loadings tell you what the dimension is "about" in the positive direction, and the words with the highest negative loadings tell you what it captures in the negative direction. A dimension might show "music, melody, harmony, song, rhythm" on one pole and "mathematics, equation, proof, theorem, algebra" on the other, suggesting it captures a technical-versus-aesthetic contrast. Another dimension might show "happy, joyful, delighted, cheerful" on one pole and "sad, gloomy, miserable, depressed" on the other, capturing an emotional valence axis.

The first dimension often captures something close to word frequency, because the most frequent words co-occur with almost everything and therefore dominate the first principal direction. This is a well-known artifact of SVD applied to count matrices. Removing or downweighting the first few dimensions can sometimes improve downstream task performance by exposing more detailed semantic structure that would otherwise be obscured by the frequency signal.

SVD dimensions are also sensitive to corpus domain. The same vocabulary represented with LSA on scientific articles will produce very different embeddings than LSA on news articles or literary fiction, because the co-occurrence statistics differ substantially across these domains. This domain sensitivity is both a limitation and a useful property. As a limitation, it means LSA embeddings do not transfer cleanly across domains: embeddings trained on medical texts may poorly represent the everyday senses of medical terms as they appear in general text. As a feature, it means embeddings can be specialized to a domain by choosing the training corpus carefully. A medical NLP system benefits from training its LSA on clinical notes, and a legal NLP system benefits from training on court documents.

Understanding which dimensions correspond to which semantic properties is a useful research exercise but not necessary for practical use. The embeddings can be used directly for cosine similarity computation, nearest-neighbor retrieval, and downstream task inputs without any interpretation of individual dimensions.

It is tempting to equate SVD dimensions with "topics," but the relationship is more complex. SVD dimensions are not topics in the sense of coherent themes with interpretable word distributions. They are mathematical abstractions that together span the semantic space of the corpus. Any individual word is represented as a weighted combination of all $k$ dimensions simultaneously. A word like "python" might have a large loading on a "programming languages" dimension, a smaller loading on a "biology/animals" dimension, and moderate loadings on several other dimensions.

The critical difference from topic models is that SVD dimensions can have negative loadings. In topic models, every word has a non-negative probability for every topic. In SVD, words can be "opposite" to a dimension: a word with a strongly negative loading on a dimension contrasts with words that have strongly positive loadings. This signed structure allows SVD to represent opposing semantic categories in a way that standard topic models cannot.