LoRA Concept › Summary

LoRA revolutionizes fine-tuning by recognizing that weight updates do not need full-rank expressiveness. By decomposing updates into the product of two small matrices $B$ and $A$, LoRA dramatically reduces trainable parameters while maintaining adaptation quality. The technique is grounded in well-supported empirical observations about the intrinsic dimensionality of fine-tuning updates, and it delivers practical benefits across memory, training speed, and storage that have made it one of the most widely adopted techniques in applied NLP.

The key concepts to remember:

• Weight decomposition: Instead of learning $\Delta W$ directly, LoRA learns $\Delta W = BA$ where $r \ll \min(d, k)$, putting the adaptation signal in a low-rank bottleneck
• Low-rank assumption: Fine-tuning updates have low intrinsic dimensionality because fine-tuning steers existing capabilities rather than building new ones, making compression nearly lossless
• Zero initialization: $B$ starts at zero, so training begins from exact pre-trained behavior and provides stability in early training
• Efficiency gains: 100-1000x fewer parameters translate to dramatically reduced memory, faster training, and compact storage that scales across many tasks
• Merge option: Adapters can be folded into base weights via $W = W_0 + \frac{\alpha}{r} BA$ for zero-overhead inference
• Flexibility: Apply to selected layers, use multiple adapters simultaneously for multi-task serving, and tune rank to match task complexity

The next chapter covers the mathematics behind LoRA's forward and backward pass in detail, including the scaling factor's role in training dynamics and theoretical properties of the low-rank constraint.