Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Singular Value Decomposition › Analyzing SVD Dimensions

Michael BrenndoerferUpdated March 21, 20263 min read

Language AI HandbookAll chapters

Part 3 of 67Distributional Semantics

Chapter 18Singular Value Decomposition › Analyzing SVD Dimensions

Reading section 11 of 14 · Analyzing SVD Dimensions · 3 minSection 11 of 14 · 3 min

Section 11 of 14 · 3 min read

Singular Value Decomposition › Analyzing SVD DimensionsOne section · Sections in this chapter11 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To ground the abstract algebra in something concrete, let us look at how SVD dimensions correspond to semantic content in practice. The approach is to examine the top-loading words for each left singular vector: the words with the most positive and most negative loadings reveal what semantic contrast the dimension is capturing.

In[22]:

Code

def top_words_for_dimension(U, s, words, dim_idx, n=5):
    """Return the top n positive and negative words for an SVD dimension."""
    loadings = U[:, dim_idx] * s[dim_idx]
    top_pos = np.argsort(loadings)[-n:][::-1]
    top_neg = np.argsort(loadings)[:n]
    return (
        [(words[i], loadings[i]) for i in top_pos],
        [(words[i], loadings[i]) for i in top_neg],
    )

Out[23]:

Console

Dimension 1 (sigma=9.327):
  Positive: software(+0.00), python(+0.00), compiler(+0.00), java(+0.00)
  Negative: bank(-5.49), water(-3.77), river(-3.77), finance(-3.77)

Dimension 2 (sigma=7.406):
  Positive: python(-0.00), algorithm(-0.00), compiler(-0.00), bank(-0.00)
  Negative: animal(-3.69), dog(-2.88), cat(-2.88), pet(-2.71)

Dimension 3 (sigma=7.369):
  Positive: animal(+0.00), dog(+0.00), cat(+0.00), feed(+0.00)
  Negative: code(-3.21), software(-3.21), java(-2.85), python(-2.85)

Dimension 4 (sigma=4.935):
  Positive: water(+1.45), money(+1.45), finance(+1.45), river(+1.45)
  Negative: bank(-3.99), compiler(-0.00), python(-0.00), feed(-0.00)

This analysis shows which words pull most strongly in each direction. On a toy corpus the interpretations are simple, but on a large corpus the same technique reveals meaningful semantic structure: frequency effects in early dimensions, domain contrasts in middle dimensions, and register or style contrasts in later dimensions.

Visualizing the SVD Components as a HeatmapLink Copied

Another way to understand the SVD is to visualize the loading matrix directly: plotting $\mathbf{U}_k$ as a heatmap with words on the rows and dimensions on the columns shows which words load most strongly on which components.

Out[24]:

Visualization