LoRA Implementation › Key Parameters

Understanding the key parameters for our LoRALinear implementation helps you make informed decisions when applying LoRA to your own models. Each parameter is a different tradeoff between expressiveness and efficiency.

The three main parameters are:

• rank: The rank $r$ of the low-rank decomposition determines the size of matrices $A$ and $B$ and thus controls the capacity of the adaptation. Lower ranks use fewer parameters and memory but may limit the complexity of adaptations the model can learn. Higher ranks give more expressiveness but increase computational cost. For most instruction-following and style-transfer tasks, ranks between 4 and 16 work well. For domain adaptation that requires new knowledge, ranks of 32 to 64 may be necessary.
• alpha: The scaling factor $\alpha$ controls the magnitude of the adaptation updates relative to the original layer outputs. Higher values make the LoRA contribution more influential, effectively amplifying the learning rate for the adaptation pathway. A common heuristic is to set $\alpha = r$ (giving a scaling of 1.0) or $\alpha = 2r$ (giving a scaling of 2.0). Setting alpha very high relative to rank can destabilize training by making the adaptation dominate the original weights.
• dropout: The probability of zeroing elements in the LoRA path during training, which helps prevent overfitting when fine-tuning on small datasets. This regularization applies only to the adaptation pathway, leaving the frozen base model unaffected. A dropout value of 0.05 to 0.1 is common for datasets with fewer than 10,000 examples; for larger datasets, you can set dropout to 0.

The interaction between rank and alpha deserves special attention. If you double the rank from $r=4$ to $r=8$ while keeping $\alpha$ fixed, the effective scaling drops from $\alpha/4$ to $\alpha/8$, halving the learning rate for the adaptation. To maintain the same effective scale, you should also double alpha. This is why many practitioners set $\alpha = r$ as a default: it keeps the scaling constant at 1.0 regardless of the chosen rank, making rank purely a capacity knob rather than a combined capacity-and-learning-rate knob.