LoRA Implementation › Key Parameters for PEFT LoraConfig

The key parameters for LoraConfig in the PEFT library give fine-grained control over the adaptation behavior. Understanding each parameter helps you make principled choices rather than relying purely on trial and error.

The main configuration parameters are:

• r: The rank of the low-rank decomposition. Lower ranks (4 to 16) use fewer parameters but may capture less complex adaptations. Higher ranks give more expressiveness at the cost of increased memory and computation. Most published LoRA papers use ranks between 4 and 64, with 8 and 16 being particularly common defaults. For instruction following tasks, $r=8$ is often sufficient. For domain adaptation with substantially different vocabulary and knowledge, $r=32$ or higher may be necessary.
• lora_alpha: The scaling factor for LoRA updates. It functions similarly to a learning rate multiplier for the adapters, controlling how strongly the adaptation influences model outputs relative to the frozen base weights. Setting lora_alpha = r gives a scaling of 1.0 (a conservative starting point). Setting lora_alpha = 2 * r gives a scaling of 2.0 (more aggressive adaptation). Most practitioners treat alpha as a fixed hyperparameter (commonly 16 or 32) rather than tuning it per experiment.
• target_modules: The specific modules to apply LoRA to. Common targets in transformers are query and value projections (q_lin, v_lin, q_proj, v_proj), which control attention patterns. Targeting more modules increases adaptation capacity but also parameter count. The optimal set depends on the task: attention projections work best for tasks requiring behavioral changes (tone, format, persona), while including feed-forward networks helps for tasks requiring new factual knowledge.
• lora_dropout: Dropout probability applied to the LoRA path to prevent overfitting during training. This is particularly important when fine-tuning on small datasets where the model might memorize rather than generalize. A dropout value of 0.05 to 0.1 is common for datasets with fewer than 10,000 examples. For larger datasets, you can reduce dropout or set it to zero.