LoRA Concept › The Weight Update Perspective

To understand LoRA, let's consider what happens inside a neural network during fine-tuning. Every pre-trained model begins its life through an extensive training process, learning patterns from massive datasets that might include books, websites, code repositories, and structured knowledge bases. This process produces a set of weight matrices that encode everything the model has learned: how to parse sentences, recognize named entities, reason about cause and effect, and generate coherent text. These weights represent the model's accumulated knowledge, frozen in numerical form.

When we fine-tune a pre-trained model, we start with weights $W_0$ learned during pre-training and update them based on our task-specific data. The fine-tuning process doesn't erase what the model already knows. Instead, it builds upon that foundation, adjusting the weights to emphasize patterns relevant to our particular task while preserving the general capabilities that make the model useful. The final fine-tuned model's weights can be written as the sum of the original weights and a task-specific update:

where:

• $W$: the final weights of the fine-tuned model, combining general and task-specific knowledge
• $W_0$: the frozen pre-trained weights, which remain static to preserve prior knowledge
• $\Delta W$: the accumulated weight update matrix, containing all learned adaptations

This equation captures a basic truth about transfer learning: the fine-tuned model is the pre-trained model plus some modification. The modification, represented by $\Delta W$, encodes everything we want the model to learn from our task-specific data. If we are training a customer service chatbot, $\Delta W$ contains the adjustments that help the model use appropriate language, understand product-specific terminology, and respond to common customer queries. If we are building a medical assistant, $\Delta W$ encodes how to interpret clinical language and apply domain knowledge appropriately.

For a weight matrix in a transformer layer with dimensions $d \times k$, this update $\Delta W$ has exactly $d \times k$ parameters, the same as the original matrix. In standard fine-tuning, every element of the weight matrix can change, which requires tracking gradients, optimizer states, and values for millions of parameters. A single attention layer's query projection in a 7B parameter model might have dimensions like $4096 \times 4096$, meaning over 16 million parameters in just one weight matrix. When you consider that a typical transformer has dozens of layers, each containing multiple weight matrices for query, key, value, and output projections, plus feed-forward networks with their own massive weight matrices, the total parameter count quickly reaches into the billions.

In full fine-tuning, we compute gradients for every element of $W$ and update them all. As we established in the previous chapter on PEFT motivation, this approach becomes prohibitively expensive for modern LLMs. The memory cost includes the weights themselves, gradients (another copy of the weights), and optimizer states (two more copies for Adam's momentum and variance estimates), giving a total memory footprint of roughly four times the weight size just for the training machinery.

Does $\Delta W$ need all those parameters to capture meaningful task adaptation? Consider what we're really doing when we fine-tune. We're not teaching the model to understand language from scratch. We're not rebuilding its world knowledge or its ability to reason. We're making relatively targeted adjustments, nudging the model to behave differently in specific ways relevant to our task. The update does not need full dimensionality. This is the central bet that LoRA makes, and the remarkable thing is that it almost always pays off.