LoRA Hyperparameters › Hyperparameter Search Strategies
Language AI HandbookAll chapters LoRA Hyperparameters › Hyperparameter Search StrategiesOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When optimal values are not clear from guidelines, systematic search can help. The key is to search efficiently, focusing computational resources on the hyperparameters most likely to impact performance. Because LoRA training is fast, hyperparameter search is much more practical than for full fine-tuning. Where evaluating one full fine-tuning configuration might take 24 hours on a cluster, evaluating a LoRA configuration might take 1-2 hours on a single GPU, which makes it feasible to evaluate dozens of configurations within a reasonable timeline.
Efficient Search Order
Not all hyperparameters deserve equal search effort. Prioritize in this order:
-
Target modules: The biggest impact on both performance and efficiency. Start with attention-only and expand if needed. This decision fundamentally changes what aspects of the model you are adapting, and getting it right matters more than precise rank selection.
-
Rank: Affects capacity directly. Search powers of 2 (4, 8, 16, 32) as these are natural scales for the problem. Try to identify the knee in the performance-vs-rank curve rather than optimizing to a specific value.
-
Alpha: Usually set relative to rank. Try , then if more adaptation strength is needed. Rarely needs to be tuned beyond these two choices.
-
Dropout: Fine-tune last. Start with 0.05, increase to 0.1 if overfitting, decrease to 0 if underfitting. The benefit is usually modest unless you are training on very limited data.
from itertools import product
def generate_search_space(priority="balanced"):
"""Generate hyperparameter search configurations."""
if priority == "efficiency":
# Minimal search for resource-constrained scenarios
ranks = [8, 16]
alpha_ratios = [1, 2]
target_sets = [["q_proj", "v_proj"]]
dropouts = [0.05]
elif priority == "balanced":
# Standard search
ranks = [8, 16, 32]
alpha_ratios = [1, 2]
target_sets = [
["q_proj", "v_proj"],
["q_proj", "k_proj", "v_proj", "o_proj"],
]
dropouts = [0.0, 0.05, 0.1]
elif priority == "thorough":
# Comprehensive search
ranks = [4, 8, 16, 32, 64]
alpha_ratios = [0.5, 1, 2]
target_sets = [
["q_proj", "v_proj"],
["q_proj", "k_proj", "v_proj", "o_proj"],
["q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "down_proj"],
]
dropouts = [0.0, 0.05, 0.1]
configs = []
for r, ratio, targets, dropout in product(
ranks, alpha_ratios, target_sets, dropouts
):
configs.append(
{
"r": r,
"lora_alpha": int(r * ratio),
"target_modules": targets,
"lora_dropout": dropout,
}
)
return configs
search_configs = {
"efficiency": generate_search_space("efficiency"),
"balanced": generate_search_space("balanced"),
"thorough": generate_search_space("thorough"),
}Search space sizes by priority:
Efficiency: 4 configurations
Balanced: 36 configurations
Thorough: 135 configurationsA balanced search with 36 configurations is manageable for most projects. Each configuration trains quickly due to LoRA's efficiency, making hyperparameter search practical even on limited hardware. Where full fine-tuning might require days to evaluate a single configuration, LoRA often allows evaluating dozens of configurations in the same timeframe.
Using Validation Performance to Guide Search
When performing hyperparameter search, a few practical principles help you extract maximum information from your experiments. First, evaluate configurations on a held-out validation set using your actual task metric (accuracy for classification, ROUGE for summarization, pass@1 for code) rather than training loss. Training loss can be misleading: a high-rank adapter might have lower training loss but higher validation loss due to overfitting.
Second, monitor training curves rather than just final values. A configuration that converges quickly to good performance is preferable to one that requires careful learning rate tuning to achieve similar results. If you observe a loss curve that drops rapidly then plateaus, the rank is likely sufficient and increasing it will not help. If the curve is still descending at the end of training, consider either more training steps or a slightly higher rank.
Third, be alert to signs of underfitting versus overfitting. A large gap between training and validation performance suggests overfitting, pointing toward lower rank, higher dropout, or lower alpha. Very similar training and validation performance that is below your target suggests underfitting, pointing toward higher rank, more target modules, or a higher alpha. This diagnostic framework helps you interpret results systematically rather than adjusting hyperparameters randomly.