LoRA Mathematics › Low-Rank Decomposition Analysis

The mathematical foundation of LoRA rests on the hypothesis that weight updates during fine-tuning have low intrinsic rank. This is a strong claim that deserves careful examination. Why should task-specific adaptations live in a low-dimensional subspace? To understand why this might be true, we can connect LoRA to concepts from matrix approximation theory and explore what empirical evidence tells us about the structure of fine-tuning updates.

The claim may seem surprising at first. Fine-tuning involves exposing a model to potentially thousands or millions of examples from a new task. How could the resulting weight changes, the accumulated product of all those gradient steps, end up in a low-dimensional subspace? The answer lies in what fine-tuning accomplishes. Pre-trained models already possess rich, general representations of language. Fine-tuning does not build these representations from scratch; it adjusts how existing representations are used. These adjustments tend to be systematic and structured rather than arbitrary, and systematic changes naturally have low rank.

Consider the analogy of a musician learning a new style. They already know how to play notes, manage rhythm, and control dynamics from years of general practice. Learning jazz requires adjusting how they apply these existing skills rather than rebuilding their musical foundation from the ground up. The adjustments are concentrated in a few key stylistic dimensions, the characteristic chord voicings, the particular approach to rhythm, the specific timbre choices, not spread uniformly across every possible parameter of their playing. Fine-tuning language models follows the same pattern, and the low-rank hypothesis captures this structure mathematically.

Recall from our discussion of SVD in Part III that any matrix can be decomposed into a sum of rank-1 components, ordered by their importance. This decomposition is the key tool for analyzing when and why low-rank approximations work well, and connecting it to LoRA reveals the theoretical guarantees behind the approach.

Any matrix $M \in \mathbb{R}^{d \times k}$ can be decomposed as:

$$M = U \Sigma V^T = \sum_{i=1}^{\min(d,k)} \sigma_i u_i v_i^T$$

where:

• $M$: the matrix to be decomposed
• $U, V$: orthogonal matrices containing singular vectors as their columns
• $\Sigma$: diagonal matrix containing the singular values $\sigma_i$ in decreasing order
• $\sigma_i$: the singular values, satisfying $\sigma_1 \geq \sigma_2 \geq \ldots \geq 0$
• $u_i, v_i$: the left and right singular vectors corresponding to $\sigma_i$

The singular values tell us how much each rank-1 component contributes to the matrix. The largest singular value $\sigma_1$ corresponds to the most important direction, the direction along which the matrix stretches inputs the most. Subsequent singular values capture progressively less important directions. A matrix whose singular values drop off quickly is one where most of its "information content" concentrates in a small number of directions, and we can discard the rest with little loss.

The Eckart-Young-Mirsky theorem makes this precise: the best rank-$r$ approximation to $M$ in the Frobenius norm is obtained by keeping only the first $r$ terms of the SVD:

$$M_r = \sum_{i=1}^{r} \sigma_i u_i v_i^T$$

where:

• $M_r$: the rank-$r$ matrix that best approximates $M$ (best in the sense of Frobenius norm)
• $r$: the target rank (number of components retained)
• $\sigma_i$: the singular values for the retained components
• $u_i, v_i$: the corresponding singular vectors

Among all possible rank-$r$ matrices, the truncated SVD provides the one closest to $M$. This is a powerful guarantee: LoRA's learned matrices $B$ and $A$ can, in principle, converge to the optimal rank-$r$ approximation of the true weight update. The approximation error is precisely:

$$\|M - M_r\|_F^2 = \sum_{i=r+1}^{\min(d,k)} \sigma_i^2$$

where:

• $\|M - M_r\|_F^2$: the squared Frobenius norm of the approximation error
• $\|\cdot\|_F$: the Frobenius norm, computed as the square root of the sum of all squared elements
• $\sigma_i$: the singular values corresponding to the discarded directions (indices $r+1$ through $\min(d,k)$)

This formula reveals when low-rank approximations work well: if the singular values decay rapidly, then the terms we discard when truncating contribute little to the total, and a low-rank approximation captures most of the matrix's structure. Conversely, if all singular values are similar in magnitude, truncation loses substantial information. The quality of LoRA's approximation is therefore fundamentally determined by the singular value spectrum of the true weight update.

Empirical studies of fine-tuned models provide encouraging evidence for LoRA's approach. When researchers compute the weight changes $\Delta W_{\text{full}} = W_{\text{finetuned}} - W_{\text{pretrained}}$ from full fine-tuning runs and examine their singular value spectra, they find rapid decay. A small number of singular values dominate while most are near zero. This suggests that the actual weight changes needed for successful fine-tuning can be well approximated by low-rank matrices, which is the central empirical motivation for the entire LoRA framework.

The effectiveness of LoRA connects to the broader concept of intrinsic dimensionality in neural network training. Despite having millions or billions of parameters, neural networks need far fewer degrees of freedom to achieve successful solutions. This phenomenon, sometimes called the "intrinsic dimensionality" of learning, has been observed across many architectures and tasks, and it provides a theoretical framework for understanding why LoRA works as well as it does.

The intrinsic dimensionality of a learning problem is the minimum number of free parameters needed to find a near-optimal solution. Li et al. (2018) showed empirically that many NLP tasks have surprisingly small intrinsic dimensions: a model could be fine-tuned successfully by constraining updates to a random subspace of just a few hundred or thousand dimensions, even in parameter spaces with millions of dimensions. This finding suggested that the apparent complexity of the parameter space is misleading, and that the "useful" part of that space is much smaller.

For fine-tuning specifically, the intuition supporting low-rank updates emerges from understanding what fine-tuning accomplishes at a mechanistic level:

1. Pre-training captures general structure: The pre-trained weights $W_0$ already encode rich representations of language. These weights have been shaped by exposure to large amounts of text, learning general patterns about syntax, semantics, world knowledge, and reasoning. These representations are highly expressive and broadly useful.
2. Fine-tuning makes targeted adjustments: Adapting to a specific task requires modifying only certain aspects of these representations. A sentiment classifier needs only to adjust how certain features map to sentiment labels, not reorganize the model's entire understanding of language. A summarization model does not need new linguistic knowledge; it needs to learn task-specific patterns about what information to preserve and condense.
3. Targeted adjustments are low-rank: These task-specific modifications can often be expressed as linear combinations of a small number of directions in weight space. If fine-tuning primarily adjusts how the model uses existing features rather than learning entirely new ones, the weight changes will have the kind of structured, correlated form that low-rank matrices can capture efficiently.

Full unconstrained fine-tuning does not produce exactly low-rank updates. Instead, low-rank approximations suffice for most downstream tasks. Full-rank updates might capture noise or provide marginal gains on very demanding tasks, but for typical applications, the low-rank constraint sacrifices little while gaining substantial efficiency. The key empirical finding is that the ratio of performance loss to parameter reduction is extremely favorable for LoRA in the regime of moderate ranks.

The rank constraint $\text{rank}(\Delta W) \leq r$ has a geometric interpretation that illuminates what LoRA learns during training. The matrix $\Delta W = BA$ maps inputs through an $r$-dimensional bottleneck, forcing all information about the input to pass through a low-dimensional representation before influencing the output. This bottleneck provides computational efficiency and constrains the geometry of the adaptation.

Information flows through the LoRA path in three stages. First, the input vector $x$ is projected down to an $r$-dimensional representation $z$ by the matrix $A$. The projection $Ax$ selects which $r$ linear combinations of the input dimensions are most relevant for the adaptation. Then, this compressed representation is expanded back to the full output dimension by the matrix $B$, which specifies how each of the $r$ dimensions should contribute to the output change. Mathematically:

$$x \xrightarrow{A} z \in \mathbb{R}^r \xrightarrow{B} \Delta h \in \mathbb{R}^d$$

where:

• $x$: the input vector carrying information about the current token or context
• $A$: the down-projection matrix that identifies which aspects of the input matter for the adaptation
• $z$: the intermediate low-rank representation, a compressed code capturing the task-relevant features
• $B$: the up-projection matrix that maps the compressed code to output changes
• $\Delta h$: the update vector in the output space, the actual change in the layer's behavior
• $r$: the rank of the adaptation, controlling the bottleneck size
• $d$: the output dimension

This bottleneck structure implies several important properties that have practical consequences. First, the update $\Delta h$ necessarily lies in the column space of $B$, which has dimension at most $r$. No matter what input $x$ we provide, the LoRA path can only produce outputs in this restricted subspace. The adaptation is confined to the directions that $B$ spans. Second, different inputs can only produce updates in this same $r$-dimensional subspace. The LoRA update cannot independently adjust every dimension of the output; it must work within the constraints of the learned subspace. This shared structure across all inputs is what makes LoRA efficient: it learns a single low-dimensional adaptation rather than per-input adjustments. Third, and importantly, the subspace is learned during training, not predetermined. The optimization process discovers which directions in weight space are most useful for the task at hand, choosing the $r$-dimensional subspace that best captures the needed adaptation.

The training process simultaneously learns two complementary aspects of the adaptation. The matrix $B$ encodes which $r$-dimensional subspace in the output space should receive updates, effectively selecting the "directions" in which the model's behavior should change. The matrix $A$ encodes how to project inputs onto this subspace, determining which aspects of the input should influence these changes and by how much. Together, $A$ and $B$ learn both the structure of the adaptation and how to apply it based on the input. This joint learning is more flexible than it might appear: even though the update lives in a fixed $r$-dimensional subspace, the input-dependent weighting provided by $A$ allows the model to apply the adaptation with different intensity and combination depending on what it is processing.