LoRA Hyperparameters › The Alpha Parameter

The scaling factor $\alpha$ (alpha) is LoRA's second key hyperparameter. While rank determines the capacity of the adaptation, alpha controls the magnitude of the adaptation's contribution to the model's output. The adapted layer output $h$ sums the frozen pre-trained projection with the scaled low-rank update. As discussed in the LoRA mathematics chapter, the formula is:

$$h = Wx + \frac{\alpha}{r} BAx$$

where:

• $h$: the output vector of the adapted layer, combining both original and adapted computations
• $W$: the frozen pre-trained weight matrix that remains unchanged throughout training
• $x$: the input vector to the layer
• $Wx$: the output from the frozen pre-trained weights, representing the model's original behavior
• $BAx$: the update computed by the low-rank adapter, representing the task-specific modification
• $\alpha$: the scaling constant, a hyperparameter you set before training
• $r$: the rank of the LoRA matrices
• $\frac{\alpha}{r}$: the scaling factor that controls adaptation strength, computed from the ratio of alpha to rank
• $B$: the up-projection matrix that expands the low-rank representation
• $A$: the down-projection matrix that compresses the input

The ratio $\frac{\alpha}{r}$ scales the LoRA output before adding it to the original transformation. This ratio is the key quantity that determines how strongly the adaptation influences the model's predictions relative to its pre-trained behavior. When $\frac{\alpha}{r} = 1$, the adapter's raw output is added directly to the frozen weights' output. When $\frac{\alpha}{r} = 2$, the adapter's contribution is doubled. When $\frac{\alpha}{r} = 0.5$, it is halved.

Think of alpha as a volume knob for the adapter. The rank determines the richness of what the adapter can say, and alpha determines how loudly it says it. A high-rank adapter with low alpha makes fine-grained but quiet suggestions to the model's computation. A low-rank adapter with high alpha makes simple but loud contributions. Both settings can be useful depending on the task, and understanding this separation helps you independently tune capacity and strength.

Alpha serves multiple purposes in the LoRA framework, each addressing a different practical concern.

Learning rate decoupling: By scaling the LoRA output, alpha allows you to use the same learning rate regardless of rank. Without this scaling, changing rank would require retuning the learning rate to achieve similar training dynamics. Consider what happens without the $\frac{\alpha}{r}$ factor: a rank-64 adapter would produce outputs with approximately 8 times the magnitude of a rank-8 adapter (due to the summation over more dimensions), completely changing the effective step size of gradient updates. The alpha scaling normalizes these differences, which makes it possible to run experiments at different ranks without adjusting the learning rate each time.

Initialization stability: The $\frac{\alpha}{r}$ factor compensates for the fact that higher ranks produce outputs with higher variance, since more terms are summed. When matrix $A$ maps the input to $r$ dimensions and matrix $B$ maps those back to the output space, the variance of the resulting transformation scales roughly with $r$. The normalization by $r$ in the denominator counteracts this growth, helping maintain consistent output magnitudes across different rank choices. This matters most early in training before the adapter has learned to produce meaningful outputs.

Adaptation strength: Practically, alpha controls how much the LoRA adaptation influences the model's output relative to the frozen pre-trained weights. A larger alpha amplifies the adapter's contribution, letting more aggressive modification of the model's behavior. A smaller alpha produces more conservative updates, keeping the model closer to its pre-trained state. For tasks where the model already has relevant capabilities, a lower alpha preserves those capabilities while letting refinement. For tasks requiring significant behavioral change, a higher alpha enables more substantial modifications.

The ratio $\frac{\alpha}{r}$ determines the effective scaling and is the quantity that directly influences training behavior. Understanding this ratio helps you reason about different configuration choices. Common configurations include:

• $\alpha = r$: The most common choice, resulting in a scaling factor of 1. The LoRA update contributes with the same magnitude as if there were no explicit scaling. This provides a neutral baseline where the raw adapter output is added directly to the pre-trained computation. Many practitioners find this works well as a default across diverse tasks.

• $\alpha = 2r$: Doubles the LoRA contribution, potentially accelerating adaptation but risking instability. With stronger updates, the model can adapt more quickly, but gradients may become large enough to cause training difficulties. This configuration is useful when you observe slow adaptation with the default scaling and want to increase the learning signal without changing the optimizer configuration.

• $\alpha = r/2$: Halves the LoRA contribution. This provides more conservative updates. This is useful when you want the model to retain more of its pre-trained behavior, important for tasks where the original capabilities should remain intact, or when training is unstable with default settings and reducing the adaptation strength can restore smooth convergence.

Selecting alpha in practice involves balancing several considerations. Here are guidelines based on common scenarios.

Default choice ($\alpha = r$): Start here. It provides balanced adaptation and has been validated across many tasks. The scaling factor of 1 means the adapter's influence is neither amplified nor diminished, giving you a clean baseline from which to adjust. Most published LoRA results use this setting as it requires no additional justification.

Higher alpha ($\alpha > r$): Consider when you need faster adaptation with limited training steps, as stronger updates allow more learning per iteration. Also appropriate when the task requires strong deviation from pre-trained behavior, necessitating larger modifications, or when you are using a low learning rate and want to compensate without changing the optimizer configuration. The most common variant is $\alpha = 2r$, which doubles the effective scaling and is widely used in instruction-tuning recipes.

Lower alpha ($\alpha < r$): Consider when fine-tuning on limited data where extra regularization matters. A lower alpha prevents the model from overwriting its useful pre-trained knowledge too aggressively. Also useful when you want to preserve more of the pre-trained model's behavior, which is important for tasks where the original capabilities should remain intact. If training is unstable with default settings, reducing alpha can restore smooth convergence by reducing the magnitude of parameter updates.

A common pattern in practice is to fix $\alpha = 16$ or $\alpha = 32$ regardless of rank. This means changing rank also changes the effective scaling, which can be useful: lower ranks get relatively stronger updates, partially compensating for their reduced capacity. This approach simplifies hyperparameter tuning by reducing the number of independent variables you need to consider. It also has an intuitive interpretation: you are saying "I want a fixed magnitude of adaptation" and letting the rank control the complexity of that adaptation independently.