LoRA Mathematics › Initialization Scheme

LoRA's initialization is important for training stability and convergence. How you initialize matrices $A$ and $B$ determines the model's starting point and influences the entire training trajectory. An initialization that disrupts the pre-trained model's behavior forces training to first undo the damage before making progress on the task. The standard LoRA scheme is carefully designed to avoid this problem while still so proper gradient flow from the first training step.

The standard scheme is:

• Matrix $A$: Initialized from $\mathcal{N}(0, \sigma^2)$ (Gaussian with small variance) or Kaiming initialization
• Matrix $B$: Initialized to zero

This asymmetric initialization, with $A$ random and $B$ zero, ensures that the product $BA = 0$ at the start of training. This means the model begins with its pre-trained behavior completely intact: the LoRA path contributes nothing to the output until training begins to modify the parameters. The model starts from a "clean" state that preserves all the hard-won knowledge from pre-training, and the LoRA path grows organically from zero as training proceeds.

Think of it as adding a second voice to a choir. At first, the second voice is silent, and the choir sounds exactly as it did before. Gradually, the second voice joins in, harmonizing with the original to create a richer sound tailored to the new piece. If the second voice started at full volume with random notes, it would disrupt the choir before anyone had a chance to coordinate. Starting from silence and growing gradually is far more conducive to good ensemble singing.

Starting with $\Delta W = BA = 0$ has several important advantages that connect to basic principles of transfer learning and optimization:

1. Continuity from pretraining: The model's initial behavior exactly matches the pre-trained model. This is valuable because the pre-trained model already performs well on many tasks. No "warm-up" period is needed for the model to recover from a random perturbation. From the first training step, we are refining good behavior rather than recovering from a disrupted starting point. This continuity property is especially important for large models where random perturbations can cause catastrophic forgetting of capabilities that were not directly targeted by fine-tuning.

2. Stable training dynamics: Large random initializations of both $A$ and $B$ could produce $\|BA\|$ values that significantly perturb the pre-trained representations. If the initial perturbation is large, early training might focus on undoing this damage rather than learning the task. By starting at zero, we ensure that the first training steps are devoted entirely to task-relevant adaptation, not error correction. This is particularly important because the pre-trained weights encode complex, interdependent representations that can be disrupted in unpredictable ways.

3. Gradient flow: A natural concern with zero initialization is whether gradients will flow properly. If $B = 0$ initially, will the gradient with respect to $B$ also be zero, preventing any learning? Fortunately, this is not the case. The gradient with respect to $B$ depends on the input to the LoRA layer and the upstream gradient, not on the current value of $B$. As we will derive in the gradient section below, $B$ receives non-zero gradients even when it equals zero, letting immediate learning from the first training step.

The initialization variance explains why the standard scheme works better than symmetric random initialization of both matrices. If we initialize both $A$ and $B$ randomly with entries drawn from $\mathcal{N}(0, \sigma^2)$, we can compute the expected magnitude of the product $BA$ and see why this creates a problem.

For a single entry of the product, we can compute the expected squared value by expanding the matrix multiplication and using the independence of the random entries:

$$\begin{aligned} \mathbb{E}[(BA)_{ij}^2] &= \mathbb{E}\left[\left(\sum_{l=1}^{r} B_{il} A_{lj}\right)^2\right] \\ &= \sum_{l=1}^{r} \mathbb{E}[B_{il}^2 A_{lj}^2] + \sum_{l \neq m} \mathbb{E}[B_{il} A_{lj} B_{im} A_{mj}] && \text{(expand the square)} \\ &= \sum_{l=1}^{r} \mathbb{E}[B_{il}^2] \, \mathbb{E}[A_{lj}^2] + 0 && \text{(independence and zero-mean)} \\ &= \sum_{l=1}^{r} \sigma^2 \sigma^2 && \text{(each variance equals } \sigma^2 \text{)} \\ &= r\sigma^4 \end{aligned}$$

where:

• $(BA)_{ij}$: element at row $i$, column $j$ of the product matrix
• $B_{il}, A_{lj}$: elements of the random matrices at specific positions
• $\sigma^2$: variance of the initialization distribution
• $r$: rank (the number of terms that sum in the matrix product)

The key observation is that this expected squared magnitude grows linearly with $r$. This rank dependence creates a serious problem: if we use the same initialization variance and learning rate for different ranks, higher ranks will produce larger initial perturbations and larger gradients. This would require rank-dependent learning rate adjustment to maintain consistent training dynamics across different rank experiments. Zero-initializing $B$ sidesteps this issue entirely by so the initial perturbation is exactly zero regardless of rank, making experiments with different ranks directly comparable.

For matrix $A$, the standard initialization uses variance that scales inversely with rank. This choice ensures that the intermediate representation $z = Ax$ has expected squared norm that does not grow with $r$, giving consistent gradient magnitudes regardless of rank:

$$A_{lj} \sim \mathcal{N}\left(0, \frac{1}{r}\right)$$

where:

• $A_{lj}$: element of matrix $A$ at row $l$, column $j$
• $\mathcal{N}(\mu, \sigma^2)$: normal distribution with mean $\mu$ and variance $\sigma^2$
• $r$: rank of the adaptation
• $1/r$: variance chosen so that the intermediate representation $z = Ax$ has unit-scale statistics

This scaling ensures that when gradients begin flowing and $B$ starts to move away from zero, the updates to $B$ have reasonable magnitude regardless of $r$. The variance is chosen so that the projection $Ax$ has expected squared norm that does not grow with $r$. Some implementations use Kaiming initialization instead, which was originally designed for rectified linear unit activations and sets:

$$A_{lj} \sim \mathcal{N}\left(0, \frac{2}{k}\right)$$

where:

• $A_{lj}$: element of matrix $A$
• $k$: input dimension of the layer (the "fan-in" in Kaiming terminology)

Kaiming initialization is designed to preserve the variance of activations through deep networks and is a reasonable alternative that may be preferred when you want the initial projections to have activation-aware statistics.

While zero-initialization of $B$ is standard and works well in most cases, researchers have explored alternatives that may offer advantages in certain scenarios. Understanding these alternatives helps you reason about when and why the standard scheme might be suboptimal.

• SVD initialization: Initialize $BA$ as a low-rank approximation of $W_{\text{fullfinetune}} - W_0$ from a reference full fine-tuning run. This requires running full fine-tuning once, but if you need to train many LoRA variants for different tasks or ranks, initializing from the SVD of a reference solution can accelerate convergence significantly. The idea is that the reference fine-tuning tells you roughly which directions matter, and you start LoRA already pointed in those directions rather than at zero.
• Symmetric initialization: Initialize both $A$ and $B$ with small random values, accepting the initial perturbation to pre-trained behavior. This may help when the pre-trained model's initial behavior is far from desired and you want both matrices to start adapting immediately, though it requires careful tuning of the initialization scale to avoid disrupting the pre-trained representations.
• Task-informed initialization: Use task-specific heuristics based on prior knowledge about the adaptation needed. For example, if adapting for a specific domain, initialize using SVD of domain-specific text representations. This approach requires domain knowledge but can dramatically reduce the number of gradient steps needed to reach good performance.

We will explore adaptive initialization strategies in the AdaLoRA chapter, where the initialization interacts with mechanisms that adjust rank during training based on gradient information.