AdaLoRA › Key Parameters

Configuring AdaLoRA well requires understanding how its parameters interact with each other and with the task characteristics. Poor choices can lead to either insufficient adaptation (if the target rank is too low) or wasted computation (if the initial rank is unnecessarily high).

The key parameters for AdaLoRA and their practical implications are:

• init_r: The initial rank allocated to all matrices before pruning begins. This should be set high enough that every potentially important direction has a chance to be represented, but not so high that training becomes prohibitively slow. Values between 8 and 16 are common for medium-sized models. Setting it too low risks missing important directions; setting it too high wastes memory and computation during the warmup phase.
• target_r: The final average rank to reach across all adapted matrices. This is the primary parameter controlling the total parameter budget of the final model. The ratio init_r / target_r determines the total amount of pruning. A ratio of 3x (e.g., init_r=12, target_r=4) is typical and generally leaves enough room for the algorithm to discover real heterogeneity.
• beta1 and beta2: Smoothing factors for the exponential moving averages of gradient sensitivity and importance scores respectively. Values around 0.85 are standard. Higher values reduce noise but make the algorithm less responsive to real changes in importance; lower values track changes faster but may make premature pruning decisions based on noisy gradients. In most cases, the default values work well.
• orth_reg_weight: The strength of the regularization term that enforces orthonormality of singular vectors. Values around 0.5 give a reasonable balance. Too weak and the decomposition loses its SVD-like separability, making the importance scores less reliable. Too strong and the orthonormality objective can dominate the task loss, slowing convergence or reducing final performance.
• total_step: The total number of training steps, used to calculate the pruning schedule. This must be set accurately to ensure the cubic schedule produces the desired pruning trajectory. If you set total_step higher than the actual number of training steps, pruning will not reach the target budget; if you set it lower, the budget will reach its target before training ends, and the remaining steps use the fixed final allocation.