LoRA Concept › Key Parameters

LoRA introduces a small set of hyperparameters that do not exist in full fine-tuning. Understanding what each one controls is needed for using LoRA effectively.

The two core parameters are:

• Rank $r$: The dimensionality of the low-rank adapters. This is the primary lever controlling the expressiveness-efficiency tradeoff. Lower ranks (4-8) are often sufficient for classification or simple instruction following tasks, where the model only needs to learn to format responses differently. Higher ranks (16-64) may be needed for complex generation tasks like code synthesis or mathematical reasoning, where the model needs to learn new compositional patterns. Common default values in practice are $r = 8$ or $r = 16$.

• Alpha $\alpha$: The scaling factor for the adapter output. The actual scaling applied to $\Delta W$ is $\alpha / r$, which means that if you set $\alpha = r$, the effective scaling is 1.0. A common heuristic is to set $\alpha = r$ (giving unit scaling) or $\alpha = 2r$ (giving scaling of 2.0). The intuition for this parameterization is that it decouples the rank from the effective learning rate: changing $r$ while keeping $\alpha$ fixed would change the effective scale of the update, which might require retuning the learning rate. By using the ratio $\alpha / r$, you can change $r$ without changing the effective update magnitude, simplifying hyperparameter search.

In addition to these core parameters, you also specify which weight matrices to apply LoRA to, which we discussed in the previous section. Most implementations default to adapting the query and value projections, and many practitioners find this to be a good starting point for most tasks.