Singular Value Decomposition › Code Implementation
Language AI HandbookAll chapters Singular Value Decomposition › Code ImplementationOne section · Sections in this chapter10 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us implement LSA from scratch to see each step clearly, then compare with scikit-learn's TruncatedSVD for practical use.
First, we install and import all required libraries:
Building a PPMI Matrix
We start with a small corpus to build a word-context co-occurrence matrix, then apply PPMI weighting as covered in the previous chapter. The corpus spans two distinct domains (animals and programming) and includes "bank" as a deliberately ambiguous word that bridges finance and geography:
# Small corpus covering two domains: animals and programming
corpus = [
"cat pet animal feed",
"dog pet animal feed",
"kitten cat animal",
"puppy dog animal",
"python code software program",
"java code software program",
"compiler python software",
"algorithm java code",
"bank finance money",
"bank river water",
]
# Build vocabulary
words = sorted(set(w for doc in corpus for w in doc.split()))
word_to_idx = {w: i for i, w in enumerate(words)}
vocab_size = len(words)Vocabulary size: 19 Words: ['algorithm', 'animal', 'bank', 'cat', 'code', 'compiler', 'dog', 'feed', 'finance', 'java', 'kitten', 'money', 'pet', 'program', 'puppy', 'python', 'river', 'software', 'water']
# Build co-occurrence matrix (document-level context)
# Each document is treated as a context window: all words in the same
# document are considered context for each other.
cooc = np.zeros((vocab_size, vocab_size))
for doc in corpus:
doc_words = doc.split()
for i, w1 in enumerate(doc_words):
for w2 in doc_words:
if w1 != w2:
cooc[word_to_idx[w1], word_to_idx[w2]] += 1
# Apply PPMI weighting
total = cooc.sum()
row_sums = cooc.sum(axis=1, keepdims=True)
col_sums = cooc.sum(axis=0, keepdims=True)
# PMI = log2(P(w,c) / (P(w) * P(c)))
# Small epsilon avoids log(0) for zero co-occurrences
epsilon = 1e-12
pmi = np.log2(
(cooc / total + epsilon)
/ ((row_sums / total) * (col_sums / total) + epsilon)
)
# PPMI: keep only positive PMI values
ppmi = np.maximum(pmi, 0)PPMI matrix shape: (19, 19) Sparsity (fraction zero): 82.27%
Most entries are zero. This reflects the sparsity typical of real co-occurrence matrices. PPMI has converted raw counts into meaningful association scores, and now we apply SVD to compress these into dense embeddings.
Computing SVD
We use numpy's linalg.svd to compute the full decomposition, then examine the singular value spectrum. The full_matrices=False option returns the economy (thin) SVD, which computes only components rather than the full square matrices:
# Full (thin) SVD
U, s, Vt = np.linalg.svd(ppmi, full_matrices=False)
# Fraction of variance explained by each component
# Each singular value squared is proportional to the variance it explains
variance_explained = s**2 / (s**2).sum()
cumulative_variance = variance_explained.cumsum()Top singular values and their contribution to total variance: Component Sigma Variance Cumulative ------------------------------------------------ 1 9.327 22.8% 22.8% 2 7.406 14.4% 37.3% 3 7.369 14.3% 51.5% 4 4.935 6.4% 57.9% 5 4.392 5.1% 63.0% 6 4.392 5.1% 68.0% 7 4.392 5.1% 73.1% 8 4.137 4.5% 77.6%
The first few components capture most of the variance. The rapid drop-off in singular values tells us the PPMI matrix has measurable low-rank structure: only a few latent dimensions are needed to capture the essential patterns, and the remainder is noise.
Truncated Embeddings
We form the word embeddings by projecting onto the top singular vectors, scaling each coordinate by the corresponding singular value. The scaling is important: it ensures that the distance between embeddings in the truncated space reflects the contribution of each dimension to the original matrix:
# Truncated SVD: k=2 for visualization, k=4 for richer embeddings
k = 2
U_k = U[:, :k]
s_k = s[:k]
# Word embeddings: rows of U_k * Sigma_k
# Shape: (vocab_size, k)
# Each row is the k-dimensional embedding for one word.
word_embeddings_2d = U_k * s_k
# Also compute k=4 embeddings for similarity comparison
k4 = min(4, len(s))
word_embeddings_4d = U[:, :k4] * s[:k4]2D word embeddings (U_k * Sigma_k): Word Dim 1 Dim 2 ------------------------------------ algorithm +0.000 -0.000 animal +0.000 -3.693 bank -5.487 -0.000 cat +0.000 -2.882 code +0.000 -0.000 compiler +0.000 -0.000 dog +0.000 -2.882 feed +0.000 -2.710 finance -3.771 -0.000 java +0.000 -0.000 kitten +0.000 -2.227 money -3.771 -0.000 pet +0.000 -2.710 program +0.000 -0.000 puppy +0.000 -2.227 python +0.000 -0.000 river -3.771 -0.000 software +0.000 -0.000 water -3.771 -0.000
The embedding coordinates directly reflect the semantic structure. Words from the same domain cluster together in the 2D space, and the ambiguous word "bank" should occupy an intermediate position. This reflects its membership in both the finance and nature clusters.
Visualizing the Embedding Space
A 2D scatter plot lets us see the semantic geometry directly:
Singular Value Spectrum
Visualizing the singular value decay curve helps guide the choice of and reveals how much low-rank structure the corpus contains:
The left panel shows the characteristic "elbow" shape: a few large singular values followed by a rapid drop to a flat tail. The right panel shows that just a few components explain the majority of variance in this small matrix. In a real corpus with hundreds of thousands of words, the number of components needed to reach 80% explained variance would be much larger, but the qualitative shape of both curves would be similar.
Cosine Similarity in Embedding Space
One of the most important tests for embedding quality is whether semantically similar words have high cosine similarity. Cosine similarity measures the angle between two vectors, ranging from -1 (opposite directions) to +1 (same direction), with 0 indicating orthogonality. Unlike Euclidean distance, cosine similarity is invariant to vector magnitude, which makes it reliable when comparing words that vary in frequency:
def cosine_similarity(v1, v2):
"""Compute cosine similarity between two vectors."""
norm1 = np.linalg.norm(v1)
norm2 = np.linalg.norm(v2)
if norm1 == 0 or norm2 == 0:
return 0.0
return np.dot(v1, v2) / (norm1 * norm2)
# Compare pairs using 4D embeddings for richer signal
word_pairs = [
("cat", "dog"), # Should be similar (both animals)
("python", "java"), # Should be similar (both programming)
("cat", "python"), # Should differ (different domains)
("bank", "finance"), # Should be moderately similar (financial context)
("bank", "river"), # Should be moderately similar (water context)
]
similarities = {}
for w1, w2 in word_pairs:
if w1 in word_to_idx and w2 in word_to_idx:
v1 = word_embeddings_4d[word_to_idx[w1]]
v2 = word_embeddings_4d[word_to_idx[w2]]
similarities[(w1, w2)] = cosine_similarity(v1, v2)Cosine similarities (k=4 LSA embeddings): Word Pair Similarity ------------------------------------- cat vs dog +1.0000 python vs java +1.0000 cat vs python -0.0000 bank vs finance +0.5434 bank vs river +0.5434
The similarity scores validate that LSA has captured domain structure. Within-domain pairs ("cat vs dog" and "python vs java") score much higher than cross-domain pairs ("cat vs python"). The ambiguous word "bank" shows moderate similarity to both finance-related and nature-related words, correctly reflecting its distributional ambiguity. This is LSA working as intended: the compressed representations encode the distributional patterns of the training corpus.
Using scikit-learn's TruncatedSVD
For real-world use with large sparse matrices, scikit-learn's TruncatedSVD is far more efficient than full SVD. It uses randomized SVD internally, handles sparse inputs natively, and provides variance explained metrics directly:
from scipy.sparse import csr_matrix
from sklearn.decomposition import TruncatedSVD
# Convert PPMI to sparse format (essential for large vocabularies)
ppmi_sparse = csr_matrix(ppmi)
# TruncatedSVD uses randomized SVD internally for efficiency
k = 4
svd = TruncatedSVD(n_components=k, random_state=42)
embeddings_sklearn = svd.fit_transform(ppmi_sparse)Input matrix shape: (19, 19) Embedding matrix shape: (19, 4) Total variance explained: 49.8% Per-component variance: Component 1: 20.3% Component 2: 11.0% Component 3: 10.9% Component 4: 7.6%
Note that TruncatedSVD.fit_transform returns the matrix directly: the row embeddings already incorporate the singular value scaling. The output shape of (vocab_size, k) gives one -dimensional embedding per word, ready for downstream use.
Comparing k Values
How does the choice of affect embedding quality? We can compare similarity rankings across different truncation levels to understand the tradeoff between noise reduction and semantic resolution:
k_values = [1, 2, 4, 6]
similarity_by_k = {}
for k_val in k_values:
k_actual = min(k_val, len(s))
emb = U[:, :k_actual] * s[:k_actual]
sim_dict = {}
for w1, w2 in word_pairs:
if w1 in word_to_idx and w2 in word_to_idx:
v1 = emb[word_to_idx[w1]]
v2 = emb[word_to_idx[w2]]
sim_dict[(w1, w2)] = cosine_similarity(v1, v2)
similarity_by_k[k_val] = sim_dictCosine similarity across k values: Word Pair k= 1 k= 2 k= 4 k= 6 --------------------------------------------------------- cat vs dog +1.00 +1.00 +1.00 +1.00 python vs java +1.00 +0.96 +1.00 +1.00 cat vs python +1.00 +0.11 -0.00 -0.00 bank vs finance +1.00 +1.00 +0.54 +0.41 bank vs river +1.00 +1.00 +0.54 +0.40
This comparison shows how the similarity structure evolves as we include more dimensions. With , only the dominant pattern is captured, and the full semantic picture is compressed to a single axis. With , finer distinctions emerge, and word pairs that share secondary contexts begin to show differentiated similarities.
A heatmap makes this evolution easy to see across all word pairs and all values simultaneously:
Key Parameters
The key parameters for TruncatedSVD are:
- n_components: The number of singular vectors to compute, controlling the embedding dimensionality. This is the most important parameter, and the choice follows the guidance in the section on choosing above.
- algorithm: Either
'randomized'(default, fast for large matrices) or'arpack'(exact, better for small matrices or when high precision is needed). For most NLP applications with matrices larger than a few thousand words,'randomized'is strongly preferred. - n_iter: Number of power iteration steps for the randomized algorithm. More iterations improve approximation quality at the cost of computation time. The default of 4 is sufficient for most NLP matrices.
- random_state: Random seed for reproducibility when using the randomized algorithm. Always set this to ensure consistent results across runs.