Singular Value Decomposition › Worked Example

Let us trace through a small LSA example to build concrete intuition. Consider a toy PPMI matrix for six words ("cat," "dog," "python," "java," "bank," "river") against four context words ("pet," "code," "money," "water"). A plausible set of PPMI values might look like this:

This $6 \times 4$ matrix has clear block structure. SVD decomposes it as $\mathbf{M} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^T$. Let's trace through what happens in each component.

The first singular value will be the largest. The corresponding left singular vector $\mathbf{u}_1$ will have large positive loadings for words with strong, concentrated co-occurrence patterns. Since "cat" and "dog" together account for most of the mass in the "pet" column, and "python" and "java" account for most of the mass in the "code" column, the first singular vector will roughly weight the animal and programming words equivalently (they both form strong clean blocks), while the first right singular vector $\mathbf{v}_1$ will split between "pet" and "code."

The second singular vector $\mathbf{u}_2$ captures the next most important contrast after removing the first component. With the dominant "frequent words co-occur a lot" effect removed, $\mathbf{u}_2$ likely separates the animal cluster from the programming cluster, or the financial/geographic words from both.

After truncating to $k=2$ dimensions, each word becomes a 2D point given by the corresponding row of $\mathbf{U}_2 \boldsymbol{\Sigma}_2$. The resulting geometry reflects the matrix structure:

• "Cat" and "dog" land near each other, because their PPMI profiles are nearly identical (both have high loading on "pet" and zero elsewhere).
• "Python" and "java" land near each other, for the same reason (both load heavily on "code").
• "Bank" lands in an intermediate region, because its PPMI profile is spread across "money" and "water" rather than concentrated in either animal or programming contexts.
• "River" lands near its dominant context "water," somewhat near "bank" but displaced toward the nature pole.

The 2D embedding has turned a 4-dimensional sparse representation into a 2-dimensional dense one. The key semantic groupings are preserved, and the intermediate position of "bank" correctly reflects its ambiguity across two contextual domains.

This toy example illustrates why SVD-based compression is often described as semantically meaningful by construction: the truncated dimensions keep the directions that explain most of the co-occurrence variance, and those directions correspond to the dominant thematic contrasts in the corpus.