Singular Value Decomposition › Choosing Embedding Dimensions

The most important hyperparameter in LSA is $k$, the number of singular values to retain. The tradeoff is fundamental: too few dimensions and the representation cannot capture enough nuance in the vocabulary; too many and you retain noise rather than signal. Getting $k$ right matters more than almost any other architectural choice in the LSA pipeline.

The primary diagnostic for choosing $k$ is a plot of the singular values in decreasing order, called a scree plot (the term comes from geology: "scree" is the debris that accumulates at the base of a cliff). The scree plot for a typical NLP co-occurrence matrix shows:

• A rapid initial decline through the first handful of dimensions, representing the dominant semantic themes
• A knee or elbow where the rate of decline slows substantially
• A long, gradually declining or flat tail representing noise and idiosyncratic patterns

The optimal $k$ is often near the elbow. Values beyond the elbow add little signal and increasingly mix noise into the representations. For tasks where you want distinct separation of major topics, choosing $k$ at or before the elbow is conservative and effective. For tasks where finer distinctions matter, you may benefit from going beyond the elbow.

In practice, the scree plot provides a floor on $k$ (include all components up to the elbow) but does not give a sharp ceiling. The right answer is usually obtained by evaluating downstream task performance across a range of $k$ values, which leads to the empirical approach described below.

Choosing $k$ by evaluating downstream task performance is the most reliable approach. Researchers have found consistent patterns across many experiments with NLP co-occurrence matrices:

• Small vocabularies and small corpora benefit from small $k$ (50-100), because there are fewer distinct semantic dimensions to capture and the matrix is not large enough to support reliable estimation of many components
• Large vocabularies with rich, diverse corpora can benefit from larger $k$ (300-500), especially when the target application requires distinguishing fine-grained semantic categories
• The original LSA paper found 100-300 dimensions worked well for document similarity and word analogy tasks
• Performance is often stable across a range of $k$ values, with a broad plateau rather than a sharp peak; a factor of 2 change in $k$ rarely produces dramatic differences

The key insight is that $k$ controls the bias-variance tradeoff. Small $k$ gives high bias (the embedding cannot distinguish closely related words or subtle semantic contrasts) but low variance (the representation is not affected by corpus-specific noise or sampling fluctuations). Large $k$ gives low bias but high variance (noise in the corpus leaks into the embedding dimensions). The optimal $k$ balances these two sources of error, and that balance point depends on the corpus size and the downstream task.

For practical recommendation: start with $k = 100$ as a baseline, evaluate on your target task, and try $k = 50$ and $k = 300$ to understand the sensitivity. If your corpus is small (fewer than 1 million tokens), prefer smaller $k$. If your corpus is large and your task requires fine-grained distinctions, larger $k$ is worth testing.

An alternative to the scree plot is to look at cumulative explained variance: what fraction of the total variance (sum of squared singular values) is captured by the first $k$ components? Common thresholds are 80-90% explained variance. In practice, the number of components needed to explain 80% of variance in a large NLP matrix can be surprisingly large, because language has many semantic dimensions. The cumulative variance curve often rises steeply at first and then flattens gradually, without a clean elbow. Using explained variance thresholds can push $k$ too high in large, diverse corpora.

Both the scree plot and cumulative variance are useful diagnostics, but neither should override direct measurement on your downstream task. The relationship between intrinsic dimensionality estimates and extrinsic task performance is not always predictable.