Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LoRA Concept › Worked Example: LoRA Arithmetic by Hand

Michael BrenndoerferNovember 30, 20254 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 255LoRA Concept › Worked Example: LoRA Arithmetic by Hand

Reading section 6 of 13 · Worked Example: LoRA Arithmetic by Hand · 4 minSection 6 of 13 · 4 min

Section 6 of 13 · 4 min read

LoRA Concept › Worked Example: LoRA Arithmetic by HandOne section · Sections in this chapter6 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To make the mathematics concrete, let's work through a tiny numerical example that illustrates every step of LoRA's forward computation. We will use small matrices so the arithmetic is traceable, but the same operations scale directly to production-sized weights.

Suppose we have a frozen weight matrix $W_0$ with dimensions $3 \times 3$ , a LoRA rank of $r = 1$ , and we want to see how the adapter modifies the output for a specific input.

Setup the matrices. We define:

W_0 = \begin{bmatrix} 2 & 0 & 1 \\ 0 & 3 & 0 \\ 1 & 0 & 2 \end{bmatrix}, \quad A = \begin{bmatrix} 0.5 & 0.2 & -0.1 \end{bmatrix}, \quad B = \begin{bmatrix} 0.3 \\ 0.4 \\ -0.2 \end{bmatrix}

Here $A$ has dimensions $1 \times 3$ (row vector, since $r = 1$ and $k = 3$ ) and $B$ has dimensions $3 \times 1$ (column vector, since $d = 3$ and $r = 1$ ). The input vector is:

x = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}

Compute the pre-trained contribution. We multiply $W_0$ by $x$ :

W_0 x = \begin{bmatrix} 2 \cdot 1 + 0 \cdot 1 + 1 \cdot 1 \\ 0 \cdot 1 + 3 \cdot 1 + 0 \cdot 1 \\ 1 \cdot 1 + 0 \cdot 1 + 2 \cdot 1 \end{bmatrix} = \begin{bmatrix} 3 \\ 3 \\ 3 \end{bmatrix}

Compute the LoRA contribution. We first apply $A$ to $x$ , then apply $B$ to the result. Computing $Ax$ :

Ax = \begin{bmatrix} 0.5 \cdot 1 + 0.2 \cdot 1 + (-0.1) \cdot 1 \end{bmatrix} = \begin{bmatrix} 0.6 \end{bmatrix}

This scalar is the projection of $x$ into the rank-1 subspace. Now computing $B(Ax)$ :

BAx = \begin{bmatrix} 0.3 \\ 0.4 \\ -0.2 \end{bmatrix} \cdot 0.6 = \begin{bmatrix} 0.18 \\ 0.24 \\ -0.12 \end{bmatrix}

Combine to get the final output. Adding the pre-trained contribution and the LoRA contribution:

h = W_0 x + BAx = \begin{bmatrix} 3 \\ 3 \\ 3 \end{bmatrix} + \begin{bmatrix} 0.18 \\ 0.24 \\ -0.12 \end{bmatrix} = \begin{bmatrix} 3.18 \\ 3.24 \\ 2.88 \end{bmatrix}

Interpret the result. Without the adapter, all three output dimensions receive the same value of 3. After the adapter, the first two dimensions are slightly boosted (3.18 and 3.24) while the third is slightly reduced (2.88). The adapter has broken the symmetry, introducing a task-specific pattern. In a real model, the adapter would introduce much richer patterns over a high-dimensional input, but the arithmetic is identical: project down, scale, project back up, add.

Count the parameters. The full weight matrix $W_0$ has $3 \times 3 = 9$ parameters. The LoRA decomposition uses $A$ with $1 \times 3 = 3$ parameters and $B$ with $3 \times 1 = 3$ parameters, for a total of 6 parameters. Even in this tiny example, we are using fewer parameters than the full matrix, and in realistic settings with $d = k = 4096$ and $r = 8$ , the reduction is 256-fold.

In[11]:

Code

import numpy as np

# Define the tiny example matrices
W_0 = np.array([[2, 0, 1], [0, 3, 0], [1, 0, 2]], dtype=float)
A = np.array([[0.5, 0.2, -0.1]])  # 1 x 3 (rank=1, k=3)
B = np.array([[0.3], [0.4], [-0.2]])  # 3 x 1 (d=3, rank=1)
x = np.array([[1], [1], [1]], dtype=float)  # 3 x 1 input vector

# Step 1: Pre-trained contribution
W0_x = W_0 @ x

# Step 2: LoRA contribution
Ax = A @ x  # project input to rank-1 subspace
BAx = B @ Ax  # project back to output dimension

# Step 3: Final output
h = W0_x + BAx

# Verification
delta_W = B @ A
h_equivalent = (W_0 + delta_W) @ x  # Should match h

Out[12]:

Console

Step-by-step computation:
  W_0 @ x = [3. 3. 3.]
  A @ x = 0.6000
  B @ (A @ x) = [ 0.18  0.24 -0.12]
  h = W_0x + BAx = [3.18 3.24 2.88]

Equivalence verification:
  (W_0 + BA) @ x = [3.18 3.24 2.88]
  Results match: True

Parameter counts:
  Full W_0: 9 parameters
  LoRA A + B: 3 + 3 = 6 parameters

The code confirms our hand-computed result. Both paths to the output (computing $W_0 x + BAx$ directly, or precomputing $W_0 + BA$ and applying it once) give identical results, verifying that LoRA's decomposition is mathematically equivalent to applying the full update matrix.