LoRA Mathematics: Low-Rank Adaptation Formulas & Gradients

In the previous chapter, we introduced LoRA as a parameter-efficient fine-tuning method that adapts large language models by learning low-rank updates to weight matrices. Instead of modifying the full weight matrix $W \in \mathbb{R}^{d \times k}$, LoRA learns two smaller matrices whose product represents the weight change. Now we turn to the mathematical foundations that make this approach work. Understanding LoRA's mathematics reveals why the method is both theoretically grounded and practically effective.

The formulation connects to basic concepts from linear algebra, specifically the matrix decomposition techniques from Part III of this book. You may recall from our discussions of singular value decomposition that complex matrices often carry most of their needed information in surprisingly few directions. LoRA exploits precisely this insight: if the "information content" of a weight update can be captured in a low-dimensional subspace, then we only need to learn the coordinates within that subspace rather than every individual weight change. This realization turns an intractable problem, updating billions of parameters, into a manageable one, updating millions.

Think of it like this: if you wanted to describe a detailed photograph, you could store every pixel individually. But if the photograph shows a face, most of its information lies in a few dozen principal features like the distances between eyes, the shape of the nose, and the curve of the jawline. Learning those features is far more efficient than memorizing every pixel, and the resulting description is nearly as faithful. LoRA applies the same logic to weight matrices. The task-specific "features" of a fine-tuning update often live in a low-dimensional subspace, and LoRA's factorized parameterization is designed to find and represent exactly those features.

By examining the initialization scheme, gradient flow, and rank selection criteria, you will gain the intuition needed to apply LoRA effectively and understand its variants covered in upcoming chapters. The mathematics has direct practical consequences. Every decision in LoRA's design, from why $B$ is initialized to zero rather than randomly, to why the scaling factor involves the ratio $\alpha/r$, emerges from concrete reasoning that you can verify and build on. Practitioners who understand this reasoning make better choices when tuning LoRA for new domains or architectures.

The chapter also prepares you for understanding the limitations inherent in the low-rank hypothesis. No assumption about data structure is universally true, and the claim that fine-tuning updates have low intrinsic rank has important edge cases. By understanding when and why the hypothesis holds, you will be better equipped to diagnose failures and choose appropriate ranks for difficult tasks.

LoRA reparameterizes weight updates using a factorized form that captures the essence of efficient adaptation. Rather than learning an arbitrary update to each weight matrix, LoRA constrains the update to live in a low-dimensional subspace. This constraint reflects the empirical observation that useful fine-tuning updates often have simple structure, revealing important patterns in how models adapt to new tasks.

To appreciate why this matters, consider the scale of the problem. A modern language model like LLaMA-7B has roughly 7 billion parameters. Even if you only wanted to fine-tune a fraction of the model, say the attention projection matrices, you would be looking at hundreds of millions of trainable parameters. Storing the gradients and optimizer states for these parameters requires multiple times their storage in GPU memory, quickly exhausting even high-end hardware. LoRA's factorized form side-steps this entirely: instead of treating each weight as an independent variable, it represents weight changes as the product of two small matrices, dramatically reducing the number of independent quantities to learn.

The key insight is that not all directions in weight space are equally useful for a given task. Most of the adaptation signal concentrates in a small subspace, and optimizing only within that subspace costs a fraction of the resources while sacrificing almost nothing in task performance. The mathematical structure of the factorized decomposition enforces this constraint naturally: you cannot express an arbitrary weight change as the product of small matrices, so LoRA forces the optimization to discover the most useful low-dimensional subspace.

For a pre-trained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA expresses the adapted weight as:

$$W = W_0 + \Delta W = W_0 + BA$$

where:

• $W$: the adapted weight matrix (final parameters)
• $W_0$: the pre-trained weight matrix (frozen during training)
• $\Delta W$: the weight update matrix (learned through gradient descent)
• $B \in \mathbb{R}^{d \times r}$: the up-projection matrix (trainable)
• $A \in \mathbb{R}^{r \times k}$: the down-projection matrix (trainable)
• $r \ll \min(d, k)$: the rank of the adaptation, the central hyperparameter

The key insight is that $\Delta W = BA$ has rank at most $r$. To understand why, recall that the rank of a matrix product cannot exceed the rank of either factor. Since $A$ has $r$ rows and $B$ has $r$ columns, the product $BA$ can have rank at most $r$. Even though $\Delta W$ has the same dimensions as $W_0$, potentially millions of entries, it lives in a much lower-dimensional subspace determined by the choice of $r$. This constraint is what enables LoRA's parameter efficiency: instead of learning $d \times k$ independent parameters, we learn only $r \times (d + k)$ parameters while still creating updates that span the full weight matrix dimensions.

Think of $A$ and $B$ as two collaborating translators working through a tiny shared vocabulary. Matrix $A$ compresses any input into a vocabulary of $r$ basic concepts, while matrix $B$ expands those $r$ concepts into the output space. Any adaptation we can express is forced to pass through this bottleneck of $r$ concepts, which dramatically limits the number of possible updates while still letting enormous flexibility about which $r$-dimensional subspace to occupy.

During the forward pass, the network must compute the output of each adapted layer. This computation reveals opportunities for efficient implementation and clarifies each matrix's role in the decomposition. The structure of the computation is not just mathematically correct; it is also computationally efficient in a way that becomes clear once we trace through the operations.

For an input $x \in \mathbb{R}^k$, the output $h \in \mathbb{R}^d$ is computed as:

$$\begin{aligned} h &= Wx \\ &= (W_0 + BA)x && \text{(substitute } W \text{)} \\ &= W_0 x + BA x && \text{(distribute)} \\ &= W_0 x + B(Ax) && \text{(associativity of matrix multiplication)} \end{aligned}$$

where:

• $h$: the output vector of the layer
• $W$: the full adapted weight matrix
• $x$: the input vector to the layer
• $W_0$: the frozen pre-trained weight matrix
• $B$: the trainable up-projection matrix
• $A$: the trainable down-projection matrix

The final line reveals LoRA's computational structure. The parenthesization in $B(Ax)$ is necessary for computational efficiency. The naive approach would form the full $d \times k$ matrix $BA$ explicitly and then multiply by $x$. This defeats the purpose of the low-rank decomposition because we would need to compute and store a matrix with $dk$ entries, which is exactly what we were trying to avoid.

Instead, we exploit the associativity of matrix multiplication to compute the result sequentially through a bottleneck of dimension $r$. The computation proceeds in three stages:

1. First, we compute $z = Ax$, yielding $z \in \mathbb{R}^r$. This operation involves an $r \times k$ matrix multiplying a $k$-dimensional vector, requiring $O(rk)$ operations.
2. Next, we compute $\Delta h = Bz$, yielding $\Delta h \in \mathbb{R}^d$. This involves a $d \times r$ matrix multiplying an $r$-dimensional vector, requiring $O(dr)$ operations.
3. Finally, we add the base model's output: $h = W_0 x + \Delta h$.

This sequential computation requires $O(rk + dr) = O(r(d+k))$ operations for the LoRA path, compared to $O(dk)$ if we materialized the full update matrix. When $r \ll \min(d, k)$, which is the regime where LoRA operates, this provides significant savings during both training and inference. For a typical transformer layer where $d = k = 4096$ and $r = 16$, the LoRA path requires roughly 0.8% of the operations needed to compute with a full update matrix.

The original LoRA paper introduces a scaling factor $\alpha$ to control the magnitude of the low-rank update relative to the pre-trained weights. This scaling factor plays a useful role in making LoRA practical across different rank choices and learning rate settings, and understanding it carefully will save you from common tuning mistakes.

The need for a scaling factor arises from a subtle interaction between the rank $r$ and the magnitude of the LoRA contribution. If you increase $r$ while keeping everything else fixed, the product $BA$ tends to produce larger values because there are more terms adding to each output. This means that switching from $r = 8$ to $r = 32$ would change the effective scale of the LoRA update even if the underlying "adaptation direction" remained the same. Such rank-dependent scaling behavior would make it impossible to transfer hyperparameter settings between different rank experiments, forcing expensive re-tuning every time you adjust $r$.

The scaled formulation addresses this by introducing an explicit normalization:

$$h = W_0 x + \frac{\alpha}{r} BA x$$

where:

• $h$: the output vector
• $W_0$: the pre-trained weight matrix
• $x$: the input vector
• $\alpha$: a constant scaling factor (a hyperparameter you choose)
• $r$: the rank of the LoRA matrices
• $B$: the up-projection matrix
• $A$: the down-projection matrix
• $BA$: the low-rank update matrix product

The ratio $\frac{\alpha}{r}$ serves several important purposes that become apparent once you understand how it interacts with rank changes.

• Learning rate stability: Without the scaling factor, changing the rank $r$ would fundamentally alter the magnitude of updates produced by the LoRA path. A higher rank means more terms adding to the product $BA$, which, all else being equal, would produce larger outputs. By dividing by $r$, we normalize this effect, letting the same learning rate to produce updates of similar magnitude regardless of rank choice.
• Initialization compatibility: As we will see in the initialization section below, the scaling interacts with how we initialize $A$ and $B$ to ensure consistent behavior at the start of training. The combination of proper initialization and scaling means that the model's behavior at the beginning of training does not depend sensitively on the rank choice.
• Hyperparameter transfer: Because the scaling normalizes the effect of rank, a good value of $\alpha$ often transfers across different rank choices. This simplifies hyperparameter search considerably: you can tune $\alpha$ at a lower rank where experiments are cheaper, then scale up the rank while keeping $\alpha$ fixed and expect similar behavior.

Think of $\alpha/r$ as the "volume knob" for the LoRA path. It controls how loudly the adaptation speaks relative to the frozen pre-trained output. Too large, and the LoRA update overwhelms the pretrained knowledge. Too small, and the model learns too slowly to adapt. Setting $\alpha = r$ makes the ratio equal to 1, effectively removing the normalization, while treating $\alpha$ as a separate hyperparameter decouples the volume knob from the rank choice.

In practice, many implementations set $\alpha = r$, making the scaling factor equal to 1 and effectively removing the normalization. Others treat $\alpha$ as a tunable hyperparameter, typically in the range $[8, 64]$. The choice depends on whether you want rank-independent behavior (use $\alpha/r$ scaling) or prefer to absorb any magnitude adjustments into the learning rate (set $\alpha = r$).

The mathematical foundation of LoRA rests on the hypothesis that weight updates during fine-tuning have low intrinsic rank. This is a strong claim that deserves careful examination. Why should task-specific adaptations live in a low-dimensional subspace? To understand why this might be true, we can connect LoRA to concepts from matrix approximation theory and explore what empirical evidence tells us about the structure of fine-tuning updates.

The claim may seem surprising at first. Fine-tuning involves exposing a model to potentially thousands or millions of examples from a new task. How could the resulting weight changes, the accumulated product of all those gradient steps, end up in a low-dimensional subspace? The answer lies in what fine-tuning accomplishes. Pre-trained models already possess rich, general representations of language. Fine-tuning does not build these representations from scratch; it adjusts how existing representations are used. These adjustments tend to be systematic and structured rather than arbitrary, and systematic changes naturally have low rank.

Consider the analogy of a musician learning a new style. They already know how to play notes, manage rhythm, and control dynamics from years of general practice. Learning jazz requires adjusting how they apply these existing skills rather than rebuilding their musical foundation from the ground up. The adjustments are concentrated in a few key stylistic dimensions, the characteristic chord voicings, the particular approach to rhythm, the specific timbre choices, not spread uniformly across every possible parameter of their playing. Fine-tuning language models follows the same pattern, and the low-rank hypothesis captures this structure mathematically.

Recall from our discussion of SVD in Part III that any matrix can be decomposed into a sum of rank-1 components, ordered by their importance. This decomposition is the key tool for analyzing when and why low-rank approximations work well, and connecting it to LoRA reveals the theoretical guarantees behind the approach.

Any matrix $M \in \mathbb{R}^{d \times k}$ can be decomposed as:

$$M = U \Sigma V^T = \sum_{i=1}^{\min(d,k)} \sigma_i u_i v_i^T$$

where:

• $M$: the matrix to be decomposed
• $U, V$: orthogonal matrices containing singular vectors as their columns
• $\Sigma$: diagonal matrix containing the singular values $\sigma_i$ in decreasing order
• $\sigma_i$: the singular values, satisfying $\sigma_1 \geq \sigma_2 \geq \ldots \geq 0$
• $u_i, v_i$: the left and right singular vectors corresponding to $\sigma_i$

The singular values tell us how much each rank-1 component contributes to the matrix. The largest singular value $\sigma_1$ corresponds to the most important direction, the direction along which the matrix stretches inputs the most. Subsequent singular values capture progressively less important directions. A matrix whose singular values drop off quickly is one where most of its "information content" concentrates in a small number of directions, and we can discard the rest with little loss.

The Eckart-Young-Mirsky theorem makes this precise: the best rank-$r$ approximation to $M$ in the Frobenius norm is obtained by keeping only the first $r$ terms of the SVD:

$$M_r = \sum_{i=1}^{r} \sigma_i u_i v_i^T$$

where:

• $M_r$: the rank-$r$ matrix that best approximates $M$ (best in the sense of Frobenius norm)
• $r$: the target rank (number of components retained)
• $\sigma_i$: the singular values for the retained components
• $u_i, v_i$: the corresponding singular vectors

Among all possible rank-$r$ matrices, the truncated SVD provides the one closest to $M$. This is a powerful guarantee: LoRA's learned matrices $B$ and $A$ can, in principle, converge to the optimal rank-$r$ approximation of the true weight update. The approximation error is precisely:

$$\|M - M_r\|_F^2 = \sum_{i=r+1}^{\min(d,k)} \sigma_i^2$$

where:

• $\|M - M_r\|_F^2$: the squared Frobenius norm of the approximation error
• $\|\cdot\|_F$: the Frobenius norm, computed as the square root of the sum of all squared elements
• $\sigma_i$: the singular values corresponding to the discarded directions (indices $r+1$ through $\min(d,k)$)

This formula reveals when low-rank approximations work well: if the singular values decay rapidly, then the terms we discard when truncating contribute little to the total, and a low-rank approximation captures most of the matrix's structure. Conversely, if all singular values are similar in magnitude, truncation loses substantial information. The quality of LoRA's approximation is therefore fundamentally determined by the singular value spectrum of the true weight update.

Empirical studies of fine-tuned models provide encouraging evidence for LoRA's approach. When researchers compute the weight changes $\Delta W_{\text{full}} = W_{\text{finetuned}} - W_{\text{pretrained}}$ from full fine-tuning runs and examine their singular value spectra, they find rapid decay. A small number of singular values dominate while most are near zero. This suggests that the actual weight changes needed for successful fine-tuning can be well approximated by low-rank matrices, which is the central empirical motivation for the entire LoRA framework.

The effectiveness of LoRA connects to the broader concept of intrinsic dimensionality in neural network training. Despite having millions or billions of parameters, neural networks need far fewer degrees of freedom to achieve successful solutions. This phenomenon, sometimes called the "intrinsic dimensionality" of learning, has been observed across many architectures and tasks, and it provides a theoretical framework for understanding why LoRA works as well as it does.

The intrinsic dimensionality of a learning problem is the minimum number of free parameters needed to find a near-optimal solution. Li et al. (2018) showed empirically that many NLP tasks have surprisingly small intrinsic dimensions: a model could be fine-tuned successfully by constraining updates to a random subspace of just a few hundred or thousand dimensions, even in parameter spaces with millions of dimensions. This finding suggested that the apparent complexity of the parameter space is misleading, and that the "useful" part of that space is much smaller.

For fine-tuning specifically, the intuition supporting low-rank updates emerges from understanding what fine-tuning accomplishes at a mechanistic level:

1. Pre-training captures general structure: The pre-trained weights $W_0$ already encode rich representations of language. These weights have been shaped by exposure to large amounts of text, learning general patterns about syntax, semantics, world knowledge, and reasoning. These representations are highly expressive and broadly useful.
2. Fine-tuning makes targeted adjustments: Adapting to a specific task requires modifying only certain aspects of these representations. A sentiment classifier needs only to adjust how certain features map to sentiment labels, not reorganize the model's entire understanding of language. A summarization model does not need new linguistic knowledge; it needs to learn task-specific patterns about what information to preserve and condense.
3. Targeted adjustments are low-rank: These task-specific modifications can often be expressed as linear combinations of a small number of directions in weight space. If fine-tuning primarily adjusts how the model uses existing features rather than learning entirely new ones, the weight changes will have the kind of structured, correlated form that low-rank matrices can capture efficiently.

Full unconstrained fine-tuning does not produce exactly low-rank updates. Instead, low-rank approximations suffice for most downstream tasks. Full-rank updates might capture noise or provide marginal gains on very demanding tasks, but for typical applications, the low-rank constraint sacrifices little while gaining substantial efficiency. The key empirical finding is that the ratio of performance loss to parameter reduction is extremely favorable for LoRA in the regime of moderate ranks.

The rank constraint $\text{rank}(\Delta W) \leq r$ has a geometric interpretation that illuminates what LoRA learns during training. The matrix $\Delta W = BA$ maps inputs through an $r$-dimensional bottleneck, forcing all information about the input to pass through a low-dimensional representation before influencing the output. This bottleneck provides computational efficiency and constrains the geometry of the adaptation.

Information flows through the LoRA path in three stages. First, the input vector $x$ is projected down to an $r$-dimensional representation $z$ by the matrix $A$. The projection $Ax$ selects which $r$ linear combinations of the input dimensions are most relevant for the adaptation. Then, this compressed representation is expanded back to the full output dimension by the matrix $B$, which specifies how each of the $r$ dimensions should contribute to the output change. Mathematically:

$$x \xrightarrow{A} z \in \mathbb{R}^r \xrightarrow{B} \Delta h \in \mathbb{R}^d$$

where:

• $x$: the input vector carrying information about the current token or context
• $A$: the down-projection matrix that identifies which aspects of the input matter for the adaptation
• $z$: the intermediate low-rank representation, a compressed code capturing the task-relevant features
• $B$: the up-projection matrix that maps the compressed code to output changes
• $\Delta h$: the update vector in the output space, the actual change in the layer's behavior
• $r$: the rank of the adaptation, controlling the bottleneck size
• $d$: the output dimension

This bottleneck structure implies several important properties that have practical consequences. First, the update $\Delta h$ necessarily lies in the column space of $B$, which has dimension at most $r$. No matter what input $x$ we provide, the LoRA path can only produce outputs in this restricted subspace. The adaptation is confined to the directions that $B$ spans. Second, different inputs can only produce updates in this same $r$-dimensional subspace. The LoRA update cannot independently adjust every dimension of the output; it must work within the constraints of the learned subspace. This shared structure across all inputs is what makes LoRA efficient: it learns a single low-dimensional adaptation rather than per-input adjustments. Third, and importantly, the subspace is learned during training, not predetermined. The optimization process discovers which directions in weight space are most useful for the task at hand, choosing the $r$-dimensional subspace that best captures the needed adaptation.

The training process simultaneously learns two complementary aspects of the adaptation. The matrix $B$ encodes which $r$-dimensional subspace in the output space should receive updates, effectively selecting the "directions" in which the model's behavior should change. The matrix $A$ encodes how to project inputs onto this subspace, determining which aspects of the input should influence these changes and by how much. Together, $A$ and $B$ learn both the structure of the adaptation and how to apply it based on the input. This joint learning is more flexible than it might appear: even though the update lives in a fixed $r$-dimensional subspace, the input-dependent weighting provided by $A$ allows the model to apply the adaptation with different intensity and combination depending on what it is processing.

Choosing the rank $r$ is one of the most important decisions when applying LoRA, and it involves balancing expressiveness against efficiency. Rank is a primary hyperparameter in LoRA, and understanding its trade-offs is needed for practical application. Lower ranks use fewer parameters and computation but constrain the adaptation's capacity to represent complex changes to the model's behavior. The right rank depends on the task, the model size, the dataset size, and the computational budget.

One useful mental model: think of rank as the number of "channels" through which fine-tuning signal can flow. With $r = 1$, there is only one channel, meaning the entire adaptation is described by a single direction in the input space and a single direction in the output space. Every input gets the same type of update; only the magnitude varies. With $r = 16$, there are 16 channels, allowing the adaptation to combine 16 different patterns of input sensitivity with 16 different patterns of output change. The expressiveness grows with $r$, as does the cost, and the key question is how many channels the task needs.

To understand the efficiency gains from LoRA, we need to compare the number of trainable parameters against full fine-tuning. This comparison reveals the quantitative case for LoRA and provides a principled basis for evaluating the trade-off between expressiveness and efficiency.

For a weight matrix $W_0 \in \mathbb{R}^{d \times k}$, full fine-tuning requires learning $dk$ parameters. LoRA instead introduces:

$$\text{LoRA parameters} = dr + rk = r(d + k)$$

where:

• $d$: output dimension of the layer
• $k$: input dimension of the layer
• $r$: rank of the adaptation
• $r(d+k)$: total parameters in the two LoRA matrices, $B$ with $dr$ entries and $A$ with $rk$ entries

The parameter ratio compared to full fine-tuning reveals the compression achieved:

$$\frac{r(d + k)}{dk} = \frac{r}{d} + \frac{r}{k}$$

where:

• $dk$: total parameters in the full weight matrix
• $r(d+k)$: total parameters in the LoRA adapters
• $r$: rank of the adaptation
• $d$: output dimension
• $k$: input dimension

This ratio decreases as the layer dimensions grow, meaning LoRA becomes proportionally more efficient for larger models. This is a necessary scaling property: the absolute savings from LoRA increase with model size, which is exactly the regime where parameter efficiency matters most.

For a transformer with $d = k = 4096$, typical of modern large language models, and $r = 16$:

$$\begin{aligned} \text{LoRA parameters} &= 16 \times (4096 + 4096) = 131{,}072 \\ \text{Full parameters} &= 4096 \times 4096 = 16{,}777{,}216 \\ \text{Ratio} &= \frac{131{,}072}{16{,}777{,}216} \approx 0.78\% \end{aligned}$$

where:

• $16$: the rank $r$ used in this example
• $4096$: the input and output dimensions ($d$ and $k$)
• $0.78\%$: the resulting parameter efficiency ratio

This calculation shows that LoRA with rank 16 uses less than 1% of the parameters that full fine-tuning would require for this single layer. The dramatic reduction holds across the entire model. If we apply LoRA to all attention projections (the query, key, value, and output matrices denoted $W_Q$, $W_K$, $W_V$, $W_O$) in each layer, the total trainable parameters remain a small fraction of the original model while still adapting the most important components of the transformer architecture.

The rank determines the capacity of the adaptation, controlling how expressive the learned weight changes can be. This trade-off is not merely academic: choosing too low a rank will leave performance on the table for complex tasks, while choosing too high a rank wastes memory and computation. Understanding the spectrum of rank choices helps you make principled decisions for new applications.

Different ranks are appropriate for different scenarios:

• $r = 1$: The update is a rank-1 matrix $\Delta W = ba^T$, an outer product of two vectors. This is highly constrained, representing the simplest possible non-trivial update. Every output dimension receives an update proportional to a fixed direction in the input, modulated by a single scalar. Despite this severe limitation, rank-1 adaptations sometimes suffice for simple tasks like binary classification where the model primarily needs to adjust a single decision boundary. The extreme parameter efficiency of rank-1 adapters makes them attractive for scenarios with very tight memory budgets.
• $r = 4$ to $r = 16$: These are common choices that balance efficiency and expressiveness for most NLP tasks. Many benchmarks show that ranks in this range achieve performance comparable to full fine-tuning while maintaining substantial parameter efficiency. This range represents the "sweet spot" for typical applications, where the rank is high enough to capture the multiple directions of adaptation that most tasks require, but low enough to provide strong parameter compression.
• $r = 64$ to $r = 256$: Higher capacity for complex adaptations, multi-task scenarios, or cases where lower ranks demonstrably underperform. These ranks sacrifice some efficiency for increased expressiveness, appropriate when the task demands more complex adaptations than lower ranks can represent. Tasks like instruction following or complex reasoning may benefit from higher ranks because they require coordinating many different aspects of the model's behavior simultaneously.
• $r = \min(d, k)$: Full rank, equivalent to unconstrained fine-tuning of the layer in terms of expressiveness. At this extreme, LoRA provides no parameter reduction but still maintains the structure of learning updates as a product of two matrices. This is primarily useful as a theoretical comparison point, confirming that the low-rank constraint is the source of both the efficiency and any performance limitations.

Empirically, rank selection depends on several factors that practitioners should consider carefully. Task complexity plays a measurable role: simple classification may need only $r = 8$, while complex generation or instruction following might benefit from $r = 64$ or higher. Dataset size interacts with rank in interesting ways. Smaller datasets benefit from lower ranks because the rank constraint provides implicit regularization, preventing overfitting by limiting the degrees of freedom available to the optimizer. Larger datasets can exploit higher ranks without overfitting and may show continued improvement as rank increases. Finally, different target modules often have different optimal ranks. Attention projections frequently need lower ranks than feed-forward layers, possibly because attention primarily routes information while feed-forward layers perform more complex transformations that require more expressive adaptations.

Trained LoRA adapters show an interesting pattern that reveals how the optimization process interacts with the rank constraint: even when $r$ is set relatively high, the effective rank of the learned $BA$ is often lower than the maximum possible. The optimization process tends to concentrate the adaptation into fewer dimensions than the allocated rank would allow, naturally discovering the minimum-rank representation that the task requires.

This phenomenon has both practical and theoretical implications. Practically, it suggests that you can sometimes set $r$ conservatively high and trust the optimizer to use only the dimensions it needs, though this comes at the cost of additional parameters and computation. Theoretically, it connects LoRA to the broader literature on implicit regularization in gradient descent, where the optimizer's geometry tends to prefer low-rank solutions even when higher-rank solutions are available.

The effective rank can be measured using the singular values of $BA$. One natural measure based on information theory is the spectral entropy:

$$\text{effective rank}(BA) = \exp\left(-\sum_{i} \tilde{\sigma}_i \log \tilde{\sigma}_i\right)$$

where:

• $\tilde{\sigma}_i$: the normalized singular values, defined so that $\tilde{\sigma}_i = \sigma_i / \sum_j \sigma_j$ and $\sum_i \tilde{\sigma}_i = 1$
• $\sigma_i$: the $i$-th singular value of the matrix $BA$
• $\exp(\cdot)$: exponential function that converts the entropy into a "perplexity" form, yielding an intuitive rank equivalent

This entropy-based measure, sometimes called the spectral entropy or perplexity of the singular value distribution, indicates how many singular values make substantial contributions to the matrix. If all $r$ singular values are equal, the normalized distribution is uniform and the effective rank equals $r$. If one singular value dominates, the distribution has low entropy and the effective rank approaches 1. Values between these extremes indicate intermediate concentration of the learned adaptation into a small number of directions.

Studies examining trained LoRA adapters consistently show that they often converge to solutions where only a few singular values dominate, even when the specified $r$ is much larger. This observation has two practical consequences. First, the actual intrinsic rank needed for the adaptation is often lower than the specified $r$, which means that practitioners can use moderate ranks without worrying much about wasting capacity. Second, the observation points toward potential improvements. Methods like AdaLoRA, covered in a later chapter, exploit this observation by adaptively adjusting the rank during training rather than fixing it in advance, allocating rank budget to layers where it is most needed.

LoRA's initialization is important for training stability and convergence. How you initialize matrices $A$ and $B$ determines the model's starting point and influences the entire training trajectory. An initialization that disrupts the pre-trained model's behavior forces training to first undo the damage before making progress on the task. The standard LoRA scheme is carefully designed to avoid this problem while still so proper gradient flow from the first training step.

The standard scheme is:

• Matrix $A$: Initialized from $\mathcal{N}(0, \sigma^2)$ (Gaussian with small variance) or Kaiming initialization
• Matrix $B$: Initialized to zero

This asymmetric initialization, with $A$ random and $B$ zero, ensures that the product $BA = 0$ at the start of training. This means the model begins with its pre-trained behavior completely intact: the LoRA path contributes nothing to the output until training begins to modify the parameters. The model starts from a "clean" state that preserves all the hard-won knowledge from pre-training, and the LoRA path grows organically from zero as training proceeds.

Think of it as adding a second voice to a choir. At first, the second voice is silent, and the choir sounds exactly as it did before. Gradually, the second voice joins in, harmonizing with the original to create a richer sound tailored to the new piece. If the second voice started at full volume with random notes, it would disrupt the choir before anyone had a chance to coordinate. Starting from silence and growing gradually is far more conducive to good ensemble singing.

Starting with $\Delta W = BA = 0$ has several important advantages that connect to basic principles of transfer learning and optimization:

1. Continuity from pretraining: The model's initial behavior exactly matches the pre-trained model. This is valuable because the pre-trained model already performs well on many tasks. No "warm-up" period is needed for the model to recover from a random perturbation. From the first training step, we are refining good behavior rather than recovering from a disrupted starting point. This continuity property is especially important for large models where random perturbations can cause catastrophic forgetting of capabilities that were not directly targeted by fine-tuning.

2. Stable training dynamics: Large random initializations of both $A$ and $B$ could produce $\|BA\|$ values that significantly perturb the pre-trained representations. If the initial perturbation is large, early training might focus on undoing this damage rather than learning the task. By starting at zero, we ensure that the first training steps are devoted entirely to task-relevant adaptation, not error correction. This is particularly important because the pre-trained weights encode complex, interdependent representations that can be disrupted in unpredictable ways.

3. Gradient flow: A natural concern with zero initialization is whether gradients will flow properly. If $B = 0$ initially, will the gradient with respect to $B$ also be zero, preventing any learning? Fortunately, this is not the case. The gradient with respect to $B$ depends on the input to the LoRA layer and the upstream gradient, not on the current value of $B$. As we will derive in the gradient section below, $B$ receives non-zero gradients even when it equals zero, letting immediate learning from the first training step.

The initialization variance explains why the standard scheme works better than symmetric random initialization of both matrices. If we initialize both $A$ and $B$ randomly with entries drawn from $\mathcal{N}(0, \sigma^2)$, we can compute the expected magnitude of the product $BA$ and see why this creates a problem.

For a single entry of the product, we can compute the expected squared value by expanding the matrix multiplication and using the independence of the random entries:

$$\begin{aligned} \mathbb{E}[(BA)_{ij}^2] &= \mathbb{E}\left[\left(\sum_{l=1}^{r} B_{il} A_{lj}\right)^2\right] \\ &= \sum_{l=1}^{r} \mathbb{E}[B_{il}^2 A_{lj}^2] + \sum_{l \neq m} \mathbb{E}[B_{il} A_{lj} B_{im} A_{mj}] && \text{(expand the square)} \\ &= \sum_{l=1}^{r} \mathbb{E}[B_{il}^2] \, \mathbb{E}[A_{lj}^2] + 0 && \text{(independence and zero-mean)} \\ &= \sum_{l=1}^{r} \sigma^2 \sigma^2 && \text{(each variance equals } \sigma^2 \text{)} \\ &= r\sigma^4 \end{aligned}$$

where:

• $(BA)_{ij}$: element at row $i$, column $j$ of the product matrix
• $B_{il}, A_{lj}$: elements of the random matrices at specific positions
• $\sigma^2$: variance of the initialization distribution
• $r$: rank (the number of terms that sum in the matrix product)

The key observation is that this expected squared magnitude grows linearly with $r$. This rank dependence creates a serious problem: if we use the same initialization variance and learning rate for different ranks, higher ranks will produce larger initial perturbations and larger gradients. This would require rank-dependent learning rate adjustment to maintain consistent training dynamics across different rank experiments. Zero-initializing $B$ sidesteps this issue entirely by so the initial perturbation is exactly zero regardless of rank, making experiments with different ranks directly comparable.

For matrix $A$, the standard initialization uses variance that scales inversely with rank. This choice ensures that the intermediate representation $z = Ax$ has expected squared norm that does not grow with $r$, giving consistent gradient magnitudes regardless of rank:

$$A_{lj} \sim \mathcal{N}\left(0, \frac{1}{r}\right)$$

where:

• $A_{lj}$: element of matrix $A$ at row $l$, column $j$
• $\mathcal{N}(\mu, \sigma^2)$: normal distribution with mean $\mu$ and variance $\sigma^2$
• $r$: rank of the adaptation
• $1/r$: variance chosen so that the intermediate representation $z = Ax$ has unit-scale statistics

This scaling ensures that when gradients begin flowing and $B$ starts to move away from zero, the updates to $B$ have reasonable magnitude regardless of $r$. The variance is chosen so that the projection $Ax$ has expected squared norm that does not grow with $r$. Some implementations use Kaiming initialization instead, which was originally designed for rectified linear unit activations and sets:

$$A_{lj} \sim \mathcal{N}\left(0, \frac{2}{k}\right)$$

where:

• $A_{lj}$: element of matrix $A$
• $k$: input dimension of the layer (the "fan-in" in Kaiming terminology)

Kaiming initialization is designed to preserve the variance of activations through deep networks and is a reasonable alternative that may be preferred when you want the initial projections to have activation-aware statistics.

While zero-initialization of $B$ is standard and works well in most cases, researchers have explored alternatives that may offer advantages in certain scenarios. Understanding these alternatives helps you reason about when and why the standard scheme might be suboptimal.

• SVD initialization: Initialize $BA$ as a low-rank approximation of $W_{\text{fullfinetune}} - W_0$ from a reference full fine-tuning run. This requires running full fine-tuning once, but if you need to train many LoRA variants for different tasks or ranks, initializing from the SVD of a reference solution can accelerate convergence significantly. The idea is that the reference fine-tuning tells you roughly which directions matter, and you start LoRA already pointed in those directions rather than at zero.
• Symmetric initialization: Initialize both $A$ and $B$ with small random values, accepting the initial perturbation to pre-trained behavior. This may help when the pre-trained model's initial behavior is far from desired and you want both matrices to start adapting immediately, though it requires careful tuning of the initialization scale to avoid disrupting the pre-trained representations.
• Task-informed initialization: Use task-specific heuristics based on prior knowledge about the adaptation needed. For example, if adapting for a specific domain, initialize using SVD of domain-specific text representations. This approach requires domain knowledge but can dramatically reduce the number of gradient steps needed to reach good performance.

We will explore adaptive initialization strategies in the AdaLoRA chapter, where the initialization interacts with mechanisms that adjust rank during training based on gradient information.

Gradient flow through LoRA reveals the training dynamics and connects to the optimization concepts from Part X. The gradient formulas explain why the zero initialization of $B$ does not prevent learning, how $A$ and $B$ co-evolve during training, and why the optimization objective makes LoRA training well-behaved.

Understanding gradients has practical value for LoRA. The specific structure of the gradient formulas determines which parameters learn first, how quickly the two matrices co-adapt, and what happens when you change the scaling factor or rank. Practitioners who understand gradient flow can diagnose training problems more easily and tune LoRA more effectively.

Consider the forward pass for a single layer with LoRA adaptation:

$$h = W_0 x + \frac{\alpha}{r} BA x$$

where:

• $h$: the output vector of the adapted layer
• $W_0$: the pre-trained weight matrix (frozen, no gradients computed)
• $x$: the input vector to the layer
• $\alpha$: the scaling factor
• $r$: the rank
• $B, A$: the LoRA up-projection and down-projection matrices (both trainable)

Let $\mathcal{L}$ be the loss function we are minimizing. To update $A$ and $B$ via gradient descent, we need $\frac{\partial \mathcal{L}}{\partial A}$ and $\frac{\partial \mathcal{L}}{\partial B}$. The chain rule traces how parameter changes propagate through the computation to affect the loss: a small change in $A$ changes the intermediate activation $Ax$, which changes the LoRA output $BAx$, which changes the layer output $h$, which eventually changes the loss $\mathcal{L}$.

First, let us define the upstream gradient. Let $\frac{\partial \mathcal{L}}{\partial h} = g \in \mathbb{R}^d$ be the gradient of the loss with respect to the layer's output. This gradient comes from the layers above in the network and tells us how changes in $h$ affect the loss. It is computed by standard backpropagation through the rest of the network and is the "signal" that drives adaptation in both $A$ and $B$.

For the gradient with respect to $B$, we apply the chain rule. The LoRA contribution to $h$ is $\frac{\alpha}{r} B(Ax)$. Treating the product $Ax$ as a fixed vector (since we are differentiating with respect to $B$, not $A$), the derivative of $B(Ax)$ with respect to $B$ has a clean outer product form:

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial B} &= \frac{\alpha}{r} \frac{\partial \mathcal{L}}{\partial h} \cdot \frac{\partial h}{\partial B} && \text{(chain rule)} \\ &= \frac{\alpha}{r} g (Ax)^T && \text{(derivative of linear map)} \end{aligned}$$

where:

• $\mathcal{L}$: the loss function
• $g = \frac{\partial \mathcal{L}}{\partial h}$: the upstream gradient vector from layers above
• $Ax$: the intermediate vector (input projected down to rank $r$)
• $(Ax)^T$: the transpose of the intermediate vector
• $\alpha$: the scaling factor
• $r$: the rank

The gradient of $B$ is an outer product between the upstream gradient $g$ and the intermediate representation $Ax$. Each entry of the gradient $(B)_{il}$ measures how much increasing that entry would affect the loss, which depends on how much gradient signal arrives at output dimension $i$ (captured by $g_i$) and how active the corresponding bottleneck dimension $l$ was (captured by $(Ax)_l$). Dimensions that were heavily activated and carry strong gradient signal receive large updates, while inactive or gradient-free dimensions receive small updates.

For processing batches efficiently, we extend this to matrix form:

$$\frac{\partial \mathcal{L}}{\partial B} = \frac{\alpha}{r} G (AX)^T$$

where:

• $\mathcal{L}$: loss function
• $\alpha$: scaling factor
• $r$: rank
• $G \in \mathbb{R}^{d \times n}$: matrix of upstream gradients for the batch, with one column per example
• $X \in \mathbb{R}^{k \times n}$: matrix of input vectors for the batch, with one column per example
• $A$: down-projection matrix

For the gradient with respect to $A$, we again apply the chain rule, but now we trace through both the multiplication by $B$ and the multiplication by $x$:

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial A} &= \frac{\alpha}{r} B^T \cdot \frac{\partial \mathcal{L}}{\partial h} \cdot x^T && \text{(chain rule through B)} \\ &= \frac{\alpha}{r} B^T g x^T && \text{(substitute } g \text{)} \end{aligned}$$

where:

• $B^T$: transpose of the up-projection matrix, which back-projects the upstream gradient into the bottleneck space
• $\mathcal{L}$: the loss function
• $\alpha$: the scaling factor
• $r$: the rank
• $g$: the upstream gradient vector
• $x$: the input vector to the layer
• $x^T$: the transpose of the input vector

This formula shows that the gradient with respect to $A$ involves two stages: first, the upstream gradient $g$ is projected back through $B$ using $B^T g$, which maps the output-space gradient into the bottleneck space. Then, this bottleneck-space gradient is combined with the input $x$ to form the outer product that updates $A$. In matrix form for batch processing:

$$\frac{\partial \mathcal{L}}{\partial A} = \frac{\alpha}{r} B^T G X^T$$

where:

• $B^T$: transpose of the up-projection matrix
• $G$: upstream gradient matrix (one column per batch example)
• $X^T$: transpose of the input batch matrix

Several properties of these gradients are noteworthy and reveal important aspects of LoRA's training dynamics. These properties have direct practical implications for how training progresses and what kinds of convergence behavior you should expect.

Independence from $W_0$: The gradients $\frac{\partial \mathcal{L}}{\partial A}$ and $\frac{\partial \mathcal{L}}{\partial B}$ are independent of $W_0$. Frozen weights affect the upstream gradient $g$ through their contribution to the forward pass, but they never appear explicitly in the LoRA gradient formulas. This separation enables efficient training where $W_0$ is never updated. You do not need to compute or store gradients for the much larger frozen weight matrices, which is the key source of LoRA's memory savings during training.

Coupled dynamics: Although $A$ and $B$ are separate parameters, their gradients are intimately coupled:

• $\frac{\partial \mathcal{L}}{\partial B}$ depends on $A$ through the term $Ax$
• $\frac{\partial \mathcal{L}}{\partial A}$ depends on $B$ through the term $B^T g$

This coupling means $A$ and $B$ co-evolve during training, with $A$'s current value determining $B$'s gradient and vice versa. This is similar to the dynamics in other factorized parameterizations like matrix factorization for recommendation systems, leading to interesting optimization behavior including potential saddle points and multiple stable solutions that represent the same effective weight update through different factorizations.

Zero initialization dynamics: At initialization ($B = 0$), the gradients behave asymmetrically:

• $\frac{\partial \mathcal{L}}{\partial B} = \frac{\alpha}{r} g (Ax)^T \neq 0$ (generally non-zero, since $A$ is random and $g$ is non-zero)
• $\frac{\partial \mathcal{L}}{\partial A} = \frac{\alpha}{r} B^T g x^T = 0$ (exactly zero because $B = 0$)

This asymmetry is important: $B$ updates immediately from the first gradient step, while $A$ has zero gradient initially. After the first update step, $B \neq 0$, and both matrices begin receiving non-zero gradients. The initial zero-gradient phase for $A$ is brief, lasting only a single step in theory, though in practice the gradients for $A$ remain small until $B$ has moved appreciably away from zero. This "bootstrap" phase, where $B$ adapts alone before both matrices co-evolve, is a direct consequence of the asymmetric initialization.

LoRA's gradient computations are efficient, requiring operations proportional to the parameter count rather than the full weight matrix size. This efficiency is not accidental; it is a direct consequence of the factorized structure and is the key reason LoRA training is so much faster than full fine-tuning.

Computing $\frac{\partial \mathcal{L}}{\partial B} = \frac{\alpha}{r} g z^T$ where $z = Ax$ involves the following costs: during the forward pass, we must store $z \in \mathbb{R}^r$, adding minimal memory overhead of just $r$ numbers per training example. The gradient computation itself is an outer product between vectors of dimension $d$ and $r$, requiring $O(dr)$ operations and creating a $d \times r$ matrix that is directly usable as the gradient for $B$.

Computing $\frac{\partial \mathcal{L}}{\partial A} = \frac{\alpha}{r} B^T g x^T$ requires first computing $B^T g$, which is a matrix-vector product costing $O(dr)$ operations (transforming the $d$-dimensional upstream gradient into an $r$-dimensional signal using $B^T$), followed by forming the outer product with $x^T$, which costs $O(rk)$ operations and produces an $r \times k$ matrix that is the gradient for $A$.

The total gradient computation is therefore $O(r(d + k))$, matching the parameter count. This is essentially optimal: we need at least one operation per parameter to compute a gradient, and LoRA achieves this lower bound up to constant factors. This efficiency extends to the memory required for gradient storage, which also scales with $r(d+k)$ rather than $dk$, explaining why LoRA reduces GPU memory requirements by over 99% compared to full fine-tuning for large models.

Concrete numbers solidify the mathematical concepts. This example connects the abstract formulas to actual numerical operations you can verify by hand, making the flow from initialization through gradient computation concrete and verifiable.

We consider a small weight matrix $W_0 \in \mathbb{R}^{4 \times 3}$ with a rank $r = 2$ adaptation. The small dimensions make the calculations manageable while capturing all the needed structure of the real application.

Setup and Initialization

We start with a pre-trained weight matrix and initialize the LoRA matrices following the standard scheme. In a real application, $W_0$ would be a slice of a transformer's weight matrix, but a small example suffices to illustrate the mathematics.

$$W_0 = \begin{bmatrix} 1.0 & 0.5 & -0.3 \\ 0.2 & 1.0 & 0.4 \\ -0.1 & 0.3 & 1.0 \\ 0.5 & -0.2 & 0.1 \end{bmatrix}$$

where $W_0 \in \mathbb{R}^{4 \times 3}$ encodes the pre-trained transformation: input dimension $k = 3$, output dimension $d = 4$.

For the LoRA matrices with rank $r = 2$, we follow the asymmetric initialization scheme: $A$ gets small random values, $B$ gets zeros:

$$A = \begin{bmatrix} 0.3 & -0.5 & 0.2 \\ 0.4 & 0.1 & -0.3 \end{bmatrix}, \quad B = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix}$$

where $A \in \mathbb{R}^{2 \times 3}$ is the down-projection (rank $\times$ input) and $B \in \mathbb{R}^{4 \times 2}$ is the up-projection (output $\times$ rank). The total trainable parameters are $r(d+k) = 2(4+3) = 14$, compared to $dk = 12$ for the full matrix (a small example; for real models the savings are vastly larger).

We set $\alpha = 2$ and $r = 2$, making the scaling factor $\frac{\alpha}{r} = 1.0$ for simplicity.

Step 1: Forward Pass at Initialization

Given input $x = [1.0, 0.5, -0.2]^T$, we trace through the three stages:

Stage 1, pre-trained output: We compute $W_0 x$ by multiplying the frozen weight matrix with the input.

$$\begin{aligned} (W_0 x)_1 &= 1.0(1.0) + 0.5(0.5) + (-0.3)(-0.2) = 1.00 + 0.25 + 0.06 = 1.31 \\ (W_0 x)_2 &= 0.2(1.0) + 1.0(0.5) + 0.4(-0.2) = 0.20 + 0.50 - 0.08 = 0.62 \\ (W_0 x)_3 &= (-0.1)(1.0) + 0.3(0.5) + 1.0(-0.2) = -0.10 + 0.15 - 0.20 = -0.15 \\ (W_0 x)_4 &= 0.5(1.0) + (-0.2)(0.5) + 0.1(-0.2) = 0.50 - 0.10 - 0.02 = 0.38 \end{aligned}$$

So $W_0 x = [1.31, 0.62, -0.15, 0.38]^T$.

Stage 2, LoRA intermediate: We compute $z = Ax$, projecting the input down to the rank-2 bottleneck.

$$\begin{aligned} z_1 &= 0.3(1.0) + (-0.5)(0.5) + 0.2(-0.2) = 0.30 - 0.25 - 0.04 = 0.01 \\ z_2 &= 0.4(1.0) + 0.1(0.5) + (-0.3)(-0.2) = 0.40 + 0.05 + 0.06 = 0.51 \end{aligned}$$

So $z = [0.01, 0.51]^T$. These two numbers are the bottleneck representation: all the information the LoRA path has about the input is compressed into these two values.

Stage 3, LoRA output: We compute $\Delta h = Bz$. Since $B = 0$, this gives $\Delta h = [0, 0, 0, 0]^T$.

Stage 4, final output: Adding the pre-trained output and the (zero) LoRA output:

$$h = W_0 x + \frac{\alpha}{r} Bz = [1.31, 0.62, -0.15, 0.38]^T + 1.0 \cdot [0, 0, 0, 0]^T = [1.31, 0.62, -0.15, 0.38]^T$$

At initialization, the output equals the pre-trained output exactly. This confirms that zero-initializing $B$ preserves the pre-trained model's behavior: no disruption to the knowledge encoded in $W_0$.

Step 2: Gradient Computation at Initialization

With upstream gradient $g = [0.1, -0.2, 0.3, 0.1]^T$ from the loss and layers above, we compute gradients for both LoRA matrices.

For matrix $B$:

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial B} &= \frac{\alpha}{r} g z^T \\ &= 1.0 \cdot \begin{bmatrix} 0.1 \\ -0.2 \\ 0.3 \\ 0.1 \end{bmatrix} \begin{bmatrix} 0.01 & 0.51 \end{bmatrix} \\ &= \begin{bmatrix} 0.001 & 0.051 \\ -0.002 & -0.102 \\ 0.003 & 0.153 \\ 0.001 & 0.051 \end{bmatrix} \end{aligned}$$

where:

• $g$: the upstream gradient vector giving the error signal from the loss
• $z^T = (Ax)^T$: the transpose of the bottleneck representation computed in the forward pass
• The outer product structure means each entry of the gradient receives a signal proportional to the corresponding upstream gradient component and bottleneck activation

For matrix $A$:

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial A} &= \frac{\alpha}{r} B^T g x^T \\ &= 1.0 \cdot \begin{bmatrix} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{bmatrix} \begin{bmatrix} 0.1 \\ -0.2 \\ 0.3 \\ 0.1 \end{bmatrix} \begin{bmatrix} 1.0 & 0.5 & -0.2 \end{bmatrix} \\ &= \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix} \end{aligned}$$

where $B^T = 0$ because $B$ is currently zero, making the entire gradient for $A$ exactly zero.

As expected from our theoretical analysis, $B$ receives non-zero gradients while $A$'s gradient is zero at initialization. The gradient for $B$ is nonzero because it does not depend on the current value of $B$, only on the upstream signal $g$ and the intermediate activation $z$. The gradient for $A$ is zero because it involves $B^T$, which is currently the zero matrix.

Step 3: First Update and Bootstrap

After one gradient descent step with learning rate $\eta = 0.01$:

$$B_{\text{new}} = B - \eta \frac{\partial \mathcal{L}}{\partial B} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix} - 0.01 \begin{bmatrix} 0.001 & 0.051 \\ -0.002 & -0.102 \\ 0.003 & 0.153 \\ 0.001 & 0.051 \end{bmatrix} = \begin{bmatrix} -0.00001 & -0.00051 \\ 0.00002 & 0.00102 \\ -0.00003 & -0.00153 \\ -0.00001 & -0.00051 \end{bmatrix}$$

Now $B \neq 0$. On the next forward pass, $B^T g$ will produce a non-zero vector, and $A$ will begin receiving gradients. The bootstrap phase is complete: after just one step, both matrices are active participants in the optimization.

This example illustrates the entire LoRA training loop in miniature: the forward pass flows through the bottleneck, the backward pass produces asymmetric gradients, and the first update breaks the symmetry to enable joint optimization. All subsequent steps involve co-evolution of $A$ and $B$.

We will implement the LoRA mathematics in PyTorch. This shows the forward pass, gradient computation, and initialization. The implementation verifies our theoretical predictions empirically, connecting the mathematical derivations to runnable code that you can experiment with and extend.

The product $BA$ is exactly zero because $B$ is initialized to zero, confirming our model starts with pretrained behavior. Matrix $A$ has dimensions (rank, in_features) for the down-projection, while $B$ has dimensions (out_features, rank) for the up-projection. Their product $BA$ maintains the full weight matrix dimensions but contains only zeros at initialization, meaning the LoRA path contributes nothing to the output until training begins.

Let's trace gradients through a forward-backward pass to confirm the asymmetric gradient behavior we derived mathematically:

As predicted by our mathematical analysis, matrix $A$'s gradient is zero at initialization because $B = 0$ in the gradient formula $\frac{\partial \mathcal{L}}{\partial A} = \frac{\alpha}{r} B^T g x^T$, while $B$ receives non-zero gradients because $\frac{\partial \mathcal{L}}{\partial B} = \frac{\alpha}{r} g (Ax)^T$ depends on $A$ (which is non-zero) rather than $B$. This asymmetry demonstrates the coupled dynamics derived mathematically: matrix $A$ begins learning only after $B$ has moved away from zero, creating a brief initial phase where $B$ adapts alone before both matrices co-evolve.

Let's compute the parameter savings for a realistic scenario corresponding to a real large language model: