LoRA Implementation › Summary

This chapter transformed the mathematical concepts from the previous chapter into working implementations. The key implementation components are:

• LoRA Module Design: A LoRALinear class wraps existing linear layers, adding parallel low-rank matrices $A$ and $B$ while keeping original weights frozen. Zero-initialization of $B$ ensures the model starts identical to the pre-trained version. This gives a clean baseline and eliminating the risk of initial performance degradation.

• Asymmetric Initialization: Initializing $B$ to zeros and $A$ with Kaiming uniform is not arbitrary. It guarantees that $BA = 0$ at initialization, so the model begins training from its pre-trained state. The Kaiming initialization for $A$ gives appropriate gradient variance for learning.

• Model Integration: A recursive injection function identifies target modules (typically attention projections) and replaces them with LoRA-wrapped versions, automatically freezing base weights. The two-pass collection-then-replacement strategy avoids graph modification errors during traversal.

• Training Loop: Standard PyTorch training, but the optimizer receives only LoRA parameters. Memory savings come from reduced gradient computation and optimizer state storage. The backward pass naturally skips frozen parameters because PyTorch's autograd only tracks tensors with requires_grad=True.

• Weight Merging: At inference time, $W_{\text{merged}} = W + \frac{\alpha}{r}BA$ produces a standard model with zero overhead. The mathematical equivalence follows from the distributive property of linear transformations. Merging can be reversed by subtracting the same delta, supporting continued training after deployment.

• PEFT Library: HuggingFace's PEFT gives production-ready LoRA with LoraConfig for configuration, get_peft_model for wrapping, and save_pretrained/merge_and_unload for deployment workflows. The library handles edge cases like tied weights and integrates with the full HuggingFace ecosystem.

The next chapter examines how to select LoRA hyperparameters, including rank, alpha, target modules, and their interactions, to optimize for your specific task and computational constraints.