LoRA Mathematics › Summary

This chapter covered the mathematical foundations of LoRA in depth, connecting abstract formulas to concrete computation and intuition:

• Core formulation: $W = W_0 + \frac{\alpha}{r}BA$ expresses weight updates as a product of two smaller matrices, constraining updates to rank at most $r$. The factorized form enables efficient computation by working through the bottleneck $z = Ax$ rather than materializing the full update matrix.

• Low-rank approximation: SVD theory through the Eckart-Young-Mirsky theorem guarantees that truncated SVD gives the best rank-$r$ approximation in Frobenius norm. Empirical studies of fine-tuning updates show rapid singular value decay, confirming that the low-rank hypothesis captures real structure in how models adapt.

• Intrinsic dimensionality: Despite millions of parameters, fine-tuning solutions live in surprisingly small subspaces. Pre-training captures general representations, and fine-tuning makes targeted adjustments that are naturally low-rank. This explains why LoRA works empirically even though it is not guaranteed by theory alone.

• Rank selection: The rank $r$ controls the expressiveness-efficiency trade-off. Ranges of $r$ from 4 to 64 achieve over 100x compression while maintaining adaptation quality for typical NLP tasks. The effective rank of trained adapters is often lower than the specified $r$, suggesting the optimizer naturally discovers minimum-rank solutions.

• Initialization: $B$ starts at zero while $A$ is randomly initialized. This ensures training begins from pre-trained behavior with zero disruption. The asymmetric initialization also avoids rank-dependent perturbation magnitudes that would complicate hyperparameter transfer across different rank choices.

• Gradient flow: $A$ and $B$ co-evolve during training through coupled gradient formulas. $A$ receives zero gradients initially but learns as $B$ moves away from zero. The total gradient computation costs $O(r(d+k))$ operations, matching the parameter count and achieving optimal efficiency.

• Weight merging: Trained LoRA adapters can be exactly merged into the base weights ($W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$), eliminating any inference overhead. This makes LoRA-trained models identical in speed to the original model after deployment.

The next chapter covers practical implementation patterns for applying LoRA to transformer architectures and integrating it with existing training pipelines, including the PEFT library and common training frameworks.