Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

AdaLoRA › SVD-Based Parameterization

Michael BrenndoerferDecember 5, 202512 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 260AdaLoRA › SVD-Based Parameterization

Reading section 2 of 11 · SVD-Based Parameterization · 12 minSection 2 of 11 · 12 min

Section 2 of 11 · 12 min read

AdaLoRA › SVD-Based ParameterizationOne section · Sections in this chapter2 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The foundation of AdaLoRA is a change in how weight updates are represented. To understand why this change matters, we need to first understand what goes wrong with standard LoRA when you try to prune individual components.

Recall from our LoRA discussion that standard LoRA is weight updates as:

\Delta W = BA

where:

$B$ : the downstream projection matrix of dimension $d \times r$
$A$ : the upstream projection matrix of dimension $r \times d_{\text{in}}$

This product produces a rank- $r$ matrix, but the individual columns of $B$ and rows of $A$ are entangled: you cannot remove one column-row pair without affecting the entire product's behavior unpredictably. To understand why this entanglement poses a problem, consider what happens when you try to prune a component. If you remove the third column of $B$ and the third row of $A$ , the remaining columns and rows still interact in complex ways during the matrix multiplication. The contribution of each column-row pair depends on all the others, which makes it impossible to cleanly isolate and evaluate individual components.

Think of standard LoRA's $BA$ product as a team project where everyone's contributions are mixed together in the final report. If you want to remove one team member's contribution, you cannot simply cross out their paragraphs because their ideas are woven throughout everyone else's writing. You would need to rewrite the entire document to fairly assess what would be lost. SVD-based parameterization solves this by making each contributor's work self-contained in its own dedicated section, so that removing one section has no effect on the others.

AdaLoRA adopts an SVD-inspired parameterization that decouples these components and resolves the entanglement problem:

\Delta W = P \Lambda Q

where:

$P \in \mathbb{R}^{d \times r}$ contains orthonormal columns (left singular vectors)
$\Lambda = \text{diag}(\lambda_1, \lambda_2, \ldots, \lambda_r)$ is a diagonal matrix of singular values
$Q \in \mathbb{R}^{r \times d_{\text{in}}}$ contains orthonormal rows (right singular vectors)

This parameterization resembles a truncated SVD, though $P$ , $\Lambda$ , and $Q$ are learned rather than computed from an existing matrix. The key advantage is separability: each triplet $(P_{:,i}, \lambda_i, Q_{i,:})$ contributes independently to the weight update. This independence arises from the orthonormality constraints on $P$ and $Q$ . When the columns of $P$ are orthogonal to each other, and the rows of $Q$ are orthogonal to each other, each singular value triplet occupies its own distinct "direction" in the weight update space. Removing one triplet leaves the others completely unchanged, letting precise surgical pruning of adaptation components.

SVD Decomposition

Singular Value Decomposition expresses any matrix $M$ as $U \Sigma V^T$ , where $U$ and $V$ have orthonormal columns and $\Sigma$ is diagonal with non-negative singular values. The singular values appear in decreasing order: $\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r \geq 0$ . The largest singular values correspond to the most "important" directions in the transformation, in the sense that they capture the most variance. A truncated SVD keeping only the top $k$ singular values gives the best rank- $k$ approximation to the original matrix in the Frobenius norm sense. AdaLoRA uses a similar structure but learns the components directly through gradient descent, letting negative singular values and soft rather than exact orthonormality. This makes the parameterization more flexible than a true SVD while retaining the key separability property.

The contribution of the $i$ -th component to the weight update can be written as an outer product scaled by the singular value:

\Delta W_i = \lambda_i \cdot P_{:,i} \cdot Q_{i,:}

where:

$\Delta W_i \in \mathbb{R}^{d \times d_{\text{in}}}$ : the rank-1 matrix contributed by the $i$ -th triplet
$\lambda_i \in \mathbb{R}$ : the scalar singular value controlling this component's magnitude
$P_{:,i} \in \mathbb{R}^{d}$ : the $i$ -th column of $P$ , a unit vector in the output space
$Q_{i,:} \in \mathbb{R}^{d_{\text{in}}}$ : the $i$ -th row of $Q$ , a unit vector in the input space

The outer product $P_{:,i} \cdot Q_{i,:}$ creates a rank-1 matrix where each entry $(a, b)$ equals $P_{a,i} \cdot Q_{i,b}$ . This matrix captures one particular "pattern" of influence: how the input direction captured by $Q_{i,:}$ maps to the output direction captured by $P_{:,i}$ . Multiplying by the scalar $\lambda_i$ scales the entire pattern's magnitude. This formulation makes pruning straightforward: setting $\lambda_i = 0$ completely removes this component from the update without affecting the others. The total weight update is the sum of all rank-1 contributions:

\Delta W = \sum_{i=1}^{r} \Delta W_i = \sum_{i=1}^{r} \lambda_i \cdot P_{:,i} \cdot Q_{i,:}

This is exactly how SVD works conceptually: the full matrix is reconstructed as a weighted sum of rank-1 outer products, with the singular values serving as the weights. Components with larger $|\lambda_i|$ contribute more to the final update; components with $\lambda_i = 0$ contribute nothing at all. This property is what makes importance-based pruning tractable: we can evaluate and remove individual components rather than analyzing the entire matrix structure.

Out[3]:

Visualization

Heatmap of the first rank-1 matrix contribution scaled by lambda=0.8, showing the dominant pattern in the weight update decomposition. — Rank-1 component from the first singular value triplet $(\lambda_1=0.8, P_{:,1}, Q_{1,:})$, the dominant contribution to the weight update.

Heatmap of the second rank-1 matrix contribution scaled by lambda=0.4, showing a moderate pattern in the weight update decomposition. — Rank-1 component from the second singular value triplet $(\lambda_2=0.4, P_{:,2}, Q_{2,:})$, a moderate contribution to the weight update.

Heatmap of the third rank-1 matrix contribution scaled by lambda=0.1, showing the smallest pattern in the weight update decomposition. — Rank-1 component from the third singular value triplet $(\lambda_3=0.1, P_{:,3}, Q_{3,:})$, the smallest contribution to the weight update.

Heatmap of the aggregated weight update matrix delta W, formed by summing all three rank-1 component matrices. — Aggregated weight update $\Delta W = \sum_i \lambda_i P_{:,i} Q_{i,:}$, the sum of all three independent rank-1 components.

The visualization above illustrates why this decomposition is so powerful. Each of the first three panels shows a single rank-1 pattern scaled by its singular value. The patterns are orthogonal to each other (they do not overlap in any systematic way), which is what the orthonormality constraints enforce. The fourth panel shows their sum. Because the components are orthogonal, their combined effect is simply additive: removing any one component produces a predictable, well-defined change in the final matrix. This is not the case in standard LoRA's $BA$ factorization, where the individual columns of $B$ are not constrained to be orthogonal and can develop complex correlations during training.

Orthonormality ConstraintsLink Copied

For the decomposition to maintain its SVD-like properties, $P$ and $Q$ should have orthonormal columns and rows respectively. Orthonormality means two things simultaneously: each column of $P$ should have unit length (normality), and different columns should be perpendicular to each other (orthogonality). The same applies to the rows of $Q$ . These constraints ensure that the singular value triplets remain independent and that the singular values $\lambda_i$ directly stand for the magnitude of each component's contribution.

If the orthonormality constraints were violated, the interpretation of the singular values would break down. A column of $P$ that is twice as long as intended would make the corresponding $\lambda_i$ appear twice as important as it is. Two correlated columns of $P$ would make their corresponding components appear redundant when they might be capturing complementary directions. The orthonormality constraints give the singular values a direct interpretive role as a measure of each component's importance.

During training, AdaLoRA enforces orthonormality approximately through a regularization term added to the training objective. Specifically, the optimizer minimizes the task loss plus a penalty for deviating from orthonormality:

R(P, Q) = \|P^T P - I\|_F^2 + \|QQ^T - I\|_F^2

where:

$R(P, Q)$ : the orthonormality regularization term
$\|\cdot\|_F$ : the Frobenius norm, which sums the squares of all matrix entries
$I$ : the identity matrix
$P^T P - I$ : the deviation of $P$ 's column Gram matrix from the identity
$QQ^T - I$ : the deviation of $Q$ 's row Gram matrix from the identity

To see why these expressions capture orthonormality, consider the matrix $P^T P$ . Its $(i,j)$ entry is $P_{:,i}^T P_{:,j}$ , the dot product of column $i$ and column $j$ . If $P^T P = I$ , then diagonal entries equal 1 (unit length) and off-diagonal entries equal 0 (mutual orthogonality). The penalty $\|P^T P - I\|_F^2$ is zero only when $P$ is perfectly column-orthonormal, and grows larger the more $P$ deviates from this ideal. Similarly, $QQ^T$ captures the dot products between different rows of $Q$ , and the penalty drives $Q$ toward row-orthonormality.

The total training objective combines the task loss with the orthogonality regularization:

\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \gamma \cdot R(P, Q)

where $\gamma$ is the orthogonality regularization weight (orth_reg_weight in the PEFT implementation, typically around 0.5). The hyperparameter $\gamma$ controls the trade-off: higher values enforce stricter orthonormality at the cost of potentially interfering with task loss minimization, while lower values allow more deviation in exchange for greater flexibility in fitting the task.

This soft constraint offers a practical trade-off between mathematical purity and computational efficiency. Hard orthonormality (using Gram-Schmidt or similar projection methods) would require expensive projections after each gradient step, fundamentally changing the optimization problem and adding significant compute overhead. The regularization approach allows the standard optimizer (typically AdamW) to handle orthonormality as just another objective, balancing it against the task loss naturally. The optimizer can temporarily violate orthonormality if doing so helps reduce the task loss, then gradually restore the constraint as training progresses. This flexibility often leads to better final solutions than rigid enforcement would allow, and it means the implementation requires no specialized optimization machinery beyond a standard gradient-based optimizer.

Out[4]:

Visualization

Heatmap of P-transpose times P without regularization, with bright off-diagonal entries showing correlated columns and diagonal values deviating from one. — $P^T P$ without regularization, showing non-unit diagonal entries and non-zero off-diagonal correlations between columns.

Heatmap of P-transpose times P with regularization, showing near-identity structure with values close to one on the diagonal and near-zero off-diagonal. — $P^T P$ with orthonormality regularization applied, showing columns pushed toward unit length and mutual orthogonality.

Heatmap of the identity matrix showing the ideal target for orthonormal columns: ones on the diagonal and zeros elsewhere. — Target identity matrix $I$ representing perfect column orthonormality, with unit diagonals and zero off-diagonals.

The visualization above shows the effect of the orthonormality regularization in practice. Without regularization, the Gram matrix $P^T P$ has off-diagonal entries that differ substantially from zero, showing that different columns of $P$ are correlated and therefore not truly independent. The diagonal entries also deviate from one, meaning the columns do not have unit length. With regularization, the Gram matrix approaches the identity: diagonal entries are close to 1 and off-diagonal entries are close to 0. The result is that the singular value triplets become independent, and the singular values acquire a direct importance interpretation.