LoRA Mathematics › Key Parameters

Understanding how the key parameters interact is needed for practical LoRA application. Each parameter has direct mathematical significance that connects to the formulas we derived:

• rank (r): The bottleneck dimension of the low-rank decomposition. Controls both expressiveness (higher rank means more capacity) and efficiency (lower rank means fewer parameters and less computation). The right rank depends on task complexity and dataset size, with typical values ranging from 4 to 64.
• alpha: Scaling factor that controls the magnitude of LoRA updates relative to the frozen pre-trained output. The ratio $\alpha/r$ acts as the "learning rate" for the LoRA path in a sense, scaling how large the learned adaptation can be. Often set equal to rank (making the ratio 1) or tuned in the range 8 to 64.
• target_modules: Which weight matrices to apply LoRA to. Applying to all attention projections (query, key, value, output) captures the most important adaptation targets in transformer models. Feed-forward layers can also benefit from LoRA but typically need different rank settings.
• initialization: Matrix $A$ uses Gaussian or Kaiming initialization to provide diverse random projections, while matrix $B$ is initialized to zero to preserve pretrained behavior and ensure stable training from the start.