LoRA Mathematics › The LoRA Formulation

LoRA reparameterizes weight updates using a factorized form that captures the essence of efficient adaptation. Rather than learning an arbitrary update to each weight matrix, LoRA constrains the update to live in a low-dimensional subspace. This constraint reflects the empirical observation that useful fine-tuning updates often have simple structure, revealing important patterns in how models adapt to new tasks.

To appreciate why this matters, consider the scale of the problem. A modern language model like LLaMA-7B has roughly 7 billion parameters. Even if you only wanted to fine-tune a fraction of the model, say the attention projection matrices, you would be looking at hundreds of millions of trainable parameters. Storing the gradients and optimizer states for these parameters requires multiple times their storage in GPU memory, quickly exhausting even high-end hardware. LoRA's factorized form side-steps this entirely: instead of treating each weight as an independent variable, it represents weight changes as the product of two small matrices, dramatically reducing the number of independent quantities to learn.

The key insight is that not all directions in weight space are equally useful for a given task. Most of the adaptation signal concentrates in a small subspace, and optimizing only within that subspace costs a fraction of the resources while sacrificing almost nothing in task performance. The mathematical structure of the factorized decomposition enforces this constraint naturally: you cannot express an arbitrary weight change as the product of small matrices, so LoRA forces the optimization to discover the most useful low-dimensional subspace.

For a pre-trained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA expresses the adapted weight as:

$$W = W_0 + \Delta W = W_0 + BA$$

where:

• $W$: the adapted weight matrix (final parameters)
• $W_0$: the pre-trained weight matrix (frozen during training)
• $\Delta W$: the weight update matrix (learned through gradient descent)
• $B \in \mathbb{R}^{d \times r}$: the up-projection matrix (trainable)
• $A \in \mathbb{R}^{r \times k}$: the down-projection matrix (trainable)
• $r \ll \min(d, k)$: the rank of the adaptation, the central hyperparameter

The key insight is that $\Delta W = BA$ has rank at most $r$. To understand why, recall that the rank of a matrix product cannot exceed the rank of either factor. Since $A$ has $r$ rows and $B$ has $r$ columns, the product $BA$ can have rank at most $r$. Even though $\Delta W$ has the same dimensions as $W_0$, potentially millions of entries, it lives in a much lower-dimensional subspace determined by the choice of $r$. This constraint is what enables LoRA's parameter efficiency: instead of learning $d \times k$ independent parameters, we learn only $r \times (d + k)$ parameters while still creating updates that span the full weight matrix dimensions.

Think of $A$ and $B$ as two collaborating translators working through a tiny shared vocabulary. Matrix $A$ compresses any input into a vocabulary of $r$ basic concepts, while matrix $B$ expands those $r$ concepts into the output space. Any adaptation we can express is forced to pass through this bottleneck of $r$ concepts, which dramatically limits the number of possible updates while still letting enormous flexibility about which $r$-dimensional subspace to occupy.

During the forward pass, the network must compute the output of each adapted layer. This computation reveals opportunities for efficient implementation and clarifies each matrix's role in the decomposition. The structure of the computation is not just mathematically correct; it is also computationally efficient in a way that becomes clear once we trace through the operations.

For an input $x \in \mathbb{R}^k$, the output $h \in \mathbb{R}^d$ is computed as:

$$\begin{aligned} h &= Wx \\ &= (W_0 + BA)x && \text{(substitute } W \text{)} \\ &= W_0 x + BA x && \text{(distribute)} \\ &= W_0 x + B(Ax) && \text{(associativity of matrix multiplication)} \end{aligned}$$

where:

• $h$: the output vector of the layer
• $W$: the full adapted weight matrix
• $x$: the input vector to the layer
• $W_0$: the frozen pre-trained weight matrix
• $B$: the trainable up-projection matrix
• $A$: the trainable down-projection matrix

The final line reveals LoRA's computational structure. The parenthesization in $B(Ax)$ is necessary for computational efficiency. The naive approach would form the full $d \times k$ matrix $BA$ explicitly and then multiply by $x$. This defeats the purpose of the low-rank decomposition because we would need to compute and store a matrix with $dk$ entries, which is exactly what we were trying to avoid.

Instead, we exploit the associativity of matrix multiplication to compute the result sequentially through a bottleneck of dimension $r$. The computation proceeds in three stages:

1. First, we compute $z = Ax$, yielding $z \in \mathbb{R}^r$. This operation involves an $r \times k$ matrix multiplying a $k$-dimensional vector, requiring $O(rk)$ operations.
2. Next, we compute $\Delta h = Bz$, yielding $\Delta h \in \mathbb{R}^d$. This involves a $d \times r$ matrix multiplying an $r$-dimensional vector, requiring $O(dr)$ operations.
3. Finally, we add the base model's output: $h = W_0 x + \Delta h$.

This sequential computation requires $O(rk + dr) = O(r(d+k))$ operations for the LoRA path, compared to $O(dk)$ if we materialized the full update matrix. When $r \ll \min(d, k)$, which is the regime where LoRA operates, this provides significant savings during both training and inference. For a typical transformer layer where $d = k = 4096$ and $r = 16$, the LoRA path requires roughly 0.8% of the operations needed to compute with a full update matrix.

The original LoRA paper introduces a scaling factor $\alpha$ to control the magnitude of the low-rank update relative to the pre-trained weights. This scaling factor plays a useful role in making LoRA practical across different rank choices and learning rate settings, and understanding it carefully will save you from common tuning mistakes.

The need for a scaling factor arises from a subtle interaction between the rank $r$ and the magnitude of the LoRA contribution. If you increase $r$ while keeping everything else fixed, the product $BA$ tends to produce larger values because there are more terms adding to each output. This means that switching from $r = 8$ to $r = 32$ would change the effective scale of the LoRA update even if the underlying "adaptation direction" remained the same. Such rank-dependent scaling behavior would make it impossible to transfer hyperparameter settings between different rank experiments, forcing expensive re-tuning every time you adjust $r$.

The scaled formulation addresses this by introducing an explicit normalization:

$$h = W_0 x + \frac{\alpha}{r} BA x$$

where:

• $h$: the output vector
• $W_0$: the pre-trained weight matrix
• $x$: the input vector
• $\alpha$: a constant scaling factor (a hyperparameter you choose)
• $r$: the rank of the LoRA matrices
• $B$: the up-projection matrix
• $A$: the down-projection matrix
• $BA$: the low-rank update matrix product

The ratio $\frac{\alpha}{r}$ serves several important purposes that become apparent once you understand how it interacts with rank changes.

• Learning rate stability: Without the scaling factor, changing the rank $r$ would fundamentally alter the magnitude of updates produced by the LoRA path. A higher rank means more terms adding to the product $BA$, which, all else being equal, would produce larger outputs. By dividing by $r$, we normalize this effect, letting the same learning rate to produce updates of similar magnitude regardless of rank choice.
• Initialization compatibility: As we will see in the initialization section below, the scaling interacts with how we initialize $A$ and $B$ to ensure consistent behavior at the start of training. The combination of proper initialization and scaling means that the model's behavior at the beginning of training does not depend sensitively on the rank choice.
• Hyperparameter transfer: Because the scaling normalizes the effect of rank, a good value of $\alpha$ often transfers across different rank choices. This simplifies hyperparameter search considerably: you can tune $\alpha$ at a lower rank where experiments are cheaper, then scale up the rank while keeping $\alpha$ fixed and expect similar behavior.

Think of $\alpha/r$ as the "volume knob" for the LoRA path. It controls how loudly the adaptation speaks relative to the frozen pre-trained output. Too large, and the LoRA update overwhelms the pretrained knowledge. Too small, and the model learns too slowly to adapt. Setting $\alpha = r$ makes the ratio equal to 1, effectively removing the normalization, while treating $\alpha$ as a separate hyperparameter decouples the volume knob from the rank choice.

In practice, many implementations set $\alpha = r$, making the scaling factor equal to 1 and effectively removing the normalization. Others treat $\alpha$ as a tunable hyperparameter, typically in the range $[8, 64]$. The choice depends on whether you want rank-independent behavior (use $\alpha/r$ scaling) or prefer to absorb any magnitude adjustments into the learning rate (set $\alpha = r$).