Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LoRA Concept › LoRA's Decomposition Strategy

Michael BrenndoerferNovember 30, 20257 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 255LoRA Concept › LoRA's Decomposition Strategy

Reading section 4 of 13 · LoRA's Decomposition Strategy · 7 minSection 4 of 13 · 7 min

Section 4 of 13 · 7 min read

LoRA Concept › LoRA's Decomposition StrategyOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Given the low-rank assumption, LoRA's approach becomes clear: instead of learning the full update matrix $\Delta W$ directly, we learn its low-rank factors. This insight transforms the fine-tuning problem from learning millions of parameters to learning thousands, while preserving the model's ability to adapt effectively.

The central structural choice is to constrain the weight update to be the product of two trainable matrices:

\Delta W = BA

where:

$\Delta W$ : the accumulated weight update matrix with dimensions $d \times k$
$B$ : the "up-projection" matrix with dimensions $d \times r$ , which expands the low-rank signal back to the full output dimension
$A$ : the "down-projection" matrix with dimensions $r \times k$ , which compresses the input to the rank- $r$ bottleneck
$r$ : the rank hyperparameter, where $r \ll \min(d, k)$ , directly controlling the expressiveness-efficiency tradeoff

The naming of these matrices reflects the information flow. When an input vector arrives at a layer, it first encounters matrix $A$ , which compresses the $k$ -dimensional input into an $r$ -dimensional intermediate representation. This compression forces the adapter to identify the most task-relevant aspects of the input, discarding information that is not useful for the current adaptation. Think of it as a forced bottleneck: only the patterns that matter most can squeeze through the narrow $r$ -dimensional passage. Matrix $B$ then takes this compressed representation and expands it back to the $d$ -dimensional output space, shaping how the compressed information influences the layer's output. The sequence of compress-then-expand is what gives LoRA its characteristic low-rank property.

During the forward pass, the modified layer computes:

\begin{aligned} h &= W_0 x + \Delta W x && \text{(apply task-specific update)} \\ &= W_0 x + BA x && \text{(substitute low-rank factors)} \end{aligned}

where:

$h$ : the output vector, containing features modified by the adapter
$W_0$ : the frozen pre-trained weight matrix, processing input using original knowledge
$x$ : the input vector, the activation arriving from the previous layer
$B$ : the trainable up-projection adapter matrix
$A$ : the trainable down-projection adapter matrix

The key insight is that this formula runs two computations in parallel and adds their outputs. The term $W_0 x$ computes the features using the frozen pre-trained knowledge, exactly as the original model would. This preserves all of the model's existing capabilities unmodified. The second term $BAx$ computes the low-rank task-specific adjustment. This adjustment operates alongside the pre-trained computation, adding a correction signal that steers the output toward task-specific behavior. Neither term cancels the other; instead, they combine additively, letting the adapter to influence the model's behavior precisely where needed while leaving unaffected regions untouched.

The original pre-trained weights $W_0$ remain frozen throughout training; only the smaller matrices $B$ and $A$ receive gradient updates. This is the core mechanism that makes LoRA so efficient. Because $W_0$ never changes, we do not need to store gradients for its millions of parameters. We do not need optimizer states like Adam's momentum and variance estimates for those parameters. The memory savings from freezing $W_0$ are enormous, and they stack multiplicatively: no gradients, no first moments, no second moments, no updated weight values to track. The reduction in trainable parameters translates directly and proportionally to reductions in gradient memory and optimizer state memory.

In[4]:

Code

import torch

# Simulate a pre-trained weight matrix (frozen)
d, k = 4096, 4096  # Typical dimensions for LLM attention projections
W_0 = torch.randn(d, k)

# LoRA parameters with rank r=8
r = 8
B = torch.randn(d, r)  # Up-projection
A = torch.randn(r, k)  # Down-projection

# The low-rank update
delta_W = B @ A

# Forward pass combines both
x = torch.randn(k, 1)  # Input vector
output = W_0 @ x + delta_W @ x

# Calculate statistics for display
params_original = d * k
params_lora = d * r + r * k
compression_ratio = params_original / params_lora

Out[5]:

Console

Original weight matrix W_0: 4096 x 4096 = 16,777,216 parameters
LoRA matrices B and A: (4096 x 8) + (8 x 4096) = 65,536 parameters
Compression ratio: 256.0x

With rank 8, we have reduced the trainable parameters by a factor of 256 compared to full fine-tuning of this single layer. The original 16 million parameters become just 65,536 LoRA parameters. This dramatic reduction occurs because the low-rank factorization exploits the redundancy we hypothesize exists in fine-tuning updates. Rather than allowing arbitrary changes across all dimensions, we constrain changes to flow through a narrow bottleneck, capturing only the most needed adaptations. Across a full 32-layer model, with multiple weight matrices adapted per layer, these compression factors multiply to produce LoRA's large practical parameter reductions.

Initialization StrategyLink Copied

The initialization of $B$ and $A$ is not a minor implementation detail; it directly determines whether training begins from a stable, well-defined state or from a chaotic one. A poor initialization might cause the adapter to produce large, arbitrary corrections from the very first forward pass, overwhelming the pre-trained signal and destabilizing early training. LoRA uses a specific initialization scheme carefully designed to avoid this problem.

The initialization rules are simple but important:

$A$ is initialized by sampling from a Gaussian distribution with small standard deviation (similar to standard weight initialization)
$B$ is initialized to all zeros

Because $B$ starts as a matrix of zeros, the product $BA$ is exactly zero regardless of what values $A$ contains. This means at the start of training, $\Delta W = BA = 0$ , and the model behaves exactly like the pre-trained model. The first forward pass produces the same outputs the pre-trained model would produce. The first predictions are exactly what the base model would predict. There are no random, potentially harmful perturbations in the earliest training steps.

Training then gradually learns the task-specific update. As gradients flow backward through the network and update both $A$ and $B$ , the product $BA$ slowly grows from zero, introducing task-specific modifications. The model's behavior shifts incrementally from pure pre-trained behavior toward task-adapted behavior. This initialization ensures we start from a known good state and make incremental modifications, never experiencing the instability that might arise from random, potentially harmful initial updates. It also means that any comparison between LoRA and full fine-tuning starts from the same baseline: both start with the pre-trained model's behavior, and we can attribute any divergence to the learning dynamics of fine-tuning rather than initialization artifacts.

Why initialize $A$ with small random values rather than zeros? If both $A$ and $B$ were initialized to zero, then $BA = 0$ at initialization (good), but the gradients of both $A$ and $B$ with respect to the loss would also be zero initially, since the output is zero and the gradient of zero with respect to either matrix through their product is structurally degenerate. By initializing $A$ with small random values, we ensure that after the first backward pass, $B$ receives meaningful gradients and can begin updating immediately. The product starts at zero (via the zero $B$ ), but the system is "primed" to start learning on the very first step.

In[6]:

Code

import torch


# LoRA initialization
def initialize_lora(d, k, r, alpha=1.0):
    """Initialize LoRA matrices following the original paper."""
    # A initialized from Gaussian, B initialized to zero
    A = torch.randn(r, k) * 0.01  # Small initialization
    B = torch.zeros(d, r)

    # Scaling factor
    scaling = alpha / r

    return A, B, scaling


A, B, scaling = initialize_lora(d=4096, k=4096, r=8, alpha=8)

# At initialization, the update is zero
delta_W_init = scaling * (B @ A)
init_norm = delta_W_init.norm().item()
init_max = delta_W_init.abs().max().item()

Out[7]:

Console

Initial delta_W norm: 0.000000
Initial delta_W max absolute value: 0.000000

The zero initialization of $B$ guarantees that training starts from the exact pre-trained model behavior. This keeps stability in early training steps. This property is especially valuable when working with large models where instability can be difficult to diagnose and correct. By starting from a known good state, we can be confident that any changes in model behavior result from meaningful learning rather than initialization artifacts.