LoRA Mathematics › Worked Example: Full Forward and Backward Pass

Concrete numbers solidify the mathematical concepts. This example connects the abstract formulas to actual numerical operations you can verify by hand, making the flow from initialization through gradient computation concrete and verifiable.

We consider a small weight matrix $W_0 \in \mathbb{R}^{4 \times 3}$ with a rank $r = 2$ adaptation. The small dimensions make the calculations manageable while capturing all the needed structure of the real application.

Setup and Initialization

We start with a pre-trained weight matrix and initialize the LoRA matrices following the standard scheme. In a real application, $W_0$ would be a slice of a transformer's weight matrix, but a small example suffices to illustrate the mathematics.

$$W_0 = \begin{bmatrix} 1.0 & 0.5 & -0.3 \\ 0.2 & 1.0 & 0.4 \\ -0.1 & 0.3 & 1.0 \\ 0.5 & -0.2 & 0.1 \end{bmatrix}$$

where $W_0 \in \mathbb{R}^{4 \times 3}$ encodes the pre-trained transformation: input dimension $k = 3$, output dimension $d = 4$.

For the LoRA matrices with rank $r = 2$, we follow the asymmetric initialization scheme: $A$ gets small random values, $B$ gets zeros:

$$A = \begin{bmatrix} 0.3 & -0.5 & 0.2 \\ 0.4 & 0.1 & -0.3 \end{bmatrix}, \quad B = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix}$$

where $A \in \mathbb{R}^{2 \times 3}$ is the down-projection (rank $\times$ input) and $B \in \mathbb{R}^{4 \times 2}$ is the up-projection (output $\times$ rank). The total trainable parameters are $r(d+k) = 2(4+3) = 14$, compared to $dk = 12$ for the full matrix (a small example; for real models the savings are vastly larger).

We set $\alpha = 2$ and $r = 2$, making the scaling factor $\frac{\alpha}{r} = 1.0$ for simplicity.

Step 1: Forward Pass at Initialization

Given input $x = [1.0, 0.5, -0.2]^T$, we trace through the three stages:

Stage 1, pre-trained output: We compute $W_0 x$ by multiplying the frozen weight matrix with the input.

$$\begin{aligned} (W_0 x)_1 &= 1.0(1.0) + 0.5(0.5) + (-0.3)(-0.2) = 1.00 + 0.25 + 0.06 = 1.31 \\ (W_0 x)_2 &= 0.2(1.0) + 1.0(0.5) + 0.4(-0.2) = 0.20 + 0.50 - 0.08 = 0.62 \\ (W_0 x)_3 &= (-0.1)(1.0) + 0.3(0.5) + 1.0(-0.2) = -0.10 + 0.15 - 0.20 = -0.15 \\ (W_0 x)_4 &= 0.5(1.0) + (-0.2)(0.5) + 0.1(-0.2) = 0.50 - 0.10 - 0.02 = 0.38 \end{aligned}$$

So $W_0 x = [1.31, 0.62, -0.15, 0.38]^T$.

Stage 2, LoRA intermediate: We compute $z = Ax$, projecting the input down to the rank-2 bottleneck.

$$\begin{aligned} z_1 &= 0.3(1.0) + (-0.5)(0.5) + 0.2(-0.2) = 0.30 - 0.25 - 0.04 = 0.01 \\ z_2 &= 0.4(1.0) + 0.1(0.5) + (-0.3)(-0.2) = 0.40 + 0.05 + 0.06 = 0.51 \end{aligned}$$

So $z = [0.01, 0.51]^T$. These two numbers are the bottleneck representation: all the information the LoRA path has about the input is compressed into these two values.

Stage 3, LoRA output: We compute $\Delta h = Bz$. Since $B = 0$, this gives $\Delta h = [0, 0, 0, 0]^T$.

Stage 4, final output: Adding the pre-trained output and the (zero) LoRA output:

$$h = W_0 x + \frac{\alpha}{r} Bz = [1.31, 0.62, -0.15, 0.38]^T + 1.0 \cdot [0, 0, 0, 0]^T = [1.31, 0.62, -0.15, 0.38]^T$$

At initialization, the output equals the pre-trained output exactly. This confirms that zero-initializing $B$ preserves the pre-trained model's behavior: no disruption to the knowledge encoded in $W_0$.

Step 2: Gradient Computation at Initialization

With upstream gradient $g = [0.1, -0.2, 0.3, 0.1]^T$ from the loss and layers above, we compute gradients for both LoRA matrices.

For matrix $B$:

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial B} &= \frac{\alpha}{r} g z^T \\ &= 1.0 \cdot \begin{bmatrix} 0.1 \\ -0.2 \\ 0.3 \\ 0.1 \end{bmatrix} \begin{bmatrix} 0.01 & 0.51 \end{bmatrix} \\ &= \begin{bmatrix} 0.001 & 0.051 \\ -0.002 & -0.102 \\ 0.003 & 0.153 \\ 0.001 & 0.051 \end{bmatrix} \end{aligned}$$

where:

• $g$: the upstream gradient vector giving the error signal from the loss
• $z^T = (Ax)^T$: the transpose of the bottleneck representation computed in the forward pass
• The outer product structure means each entry of the gradient receives a signal proportional to the corresponding upstream gradient component and bottleneck activation

For matrix $A$:

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial A} &= \frac{\alpha}{r} B^T g x^T \\ &= 1.0 \cdot \begin{bmatrix} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{bmatrix} \begin{bmatrix} 0.1 \\ -0.2 \\ 0.3 \\ 0.1 \end{bmatrix} \begin{bmatrix} 1.0 & 0.5 & -0.2 \end{bmatrix} \\ &= \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix} \end{aligned}$$

where $B^T = 0$ because $B$ is currently zero, making the entire gradient for $A$ exactly zero.

As expected from our theoretical analysis, $B$ receives non-zero gradients while $A$'s gradient is zero at initialization. The gradient for $B$ is nonzero because it does not depend on the current value of $B$, only on the upstream signal $g$ and the intermediate activation $z$. The gradient for $A$ is zero because it involves $B^T$, which is currently the zero matrix.

Step 3: First Update and Bootstrap

After one gradient descent step with learning rate $\eta = 0.01$:

$$B_{\text{new}} = B - \eta \frac{\partial \mathcal{L}}{\partial B} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix} - 0.01 \begin{bmatrix} 0.001 & 0.051 \\ -0.002 & -0.102 \\ 0.003 & 0.153 \\ 0.001 & 0.051 \end{bmatrix} = \begin{bmatrix} -0.00001 & -0.00051 \\ 0.00002 & 0.00102 \\ -0.00003 & -0.00153 \\ -0.00001 & -0.00051 \end{bmatrix}$$

Now $B \neq 0$. On the next forward pass, $B^T g$ will produce a non-zero vector, and $A$ will begin receiving gradients. The bootstrap phase is complete: after just one step, both matrices are active participants in the optimization.

This example illustrates the entire LoRA training loop in miniature: the forward pass flows through the bottleneck, the backward pass produces asymmetric gradients, and the first update breaks the symmetry to enable joint optimization. All subsequent steps involve co-evolution of $A$ and $B$.