LoRA Hyperparameters › Putting It Together: Configuration Recipes

Michael BrenndoerferDecember 3, 20255 min read
Language AI HandbookAll chapters
Part 35 of 67Parameter-Efficient Fine-tuning
Chapter 258LoRA Hyperparameters › Putting It Together: Configuration Recipes
Reading section 6 of 10 · Putting It Together: Configuration Recipes · 5 minSection 6 of 10 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 258 of 513

Section 6 of 10 · 5 min read

LoRA Hyperparameters › Putting It Together: Configuration RecipesOne section · Sections in this chapter6 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let us consolidate these guidelines into practical configurations for common scenarios. These recipes represent starting points based on accumulated practitioner experience, though optimal configurations may vary with your model and data as well as your requirements. Think of these as reasonable defaults that work well across a range of similar situations, not as universal optima.

In[22]:
Code
def get_lora_config(scenario: str) -> dict:
    """Return recommended LoRA configuration for common scenarios."""

    configs = {
        "classification_small_data": {
            "r": 8,
            "lora_alpha": 8,
            "target_modules": ["q_proj", "v_proj"],
            "lora_dropout": 0.1,
            "description": "Conservative config for limited data",
        },
        "instruction_tuning": {
            "r": 16,
            "lora_alpha": 32,
            "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"],
            "lora_dropout": 0.05,
            "description": "Balanced config for instruction following",
        },
        "domain_adaptation": {
            "r": 32,
            "lora_alpha": 64,
            "target_modules": [
                "q_proj",
                "v_proj",
                "o_proj",
                "gate_proj",
                "up_proj",
                "down_proj",
            ],
            "lora_dropout": 0.05,
            "description": "High-capacity config for domain shift",
        },
        "code_generation": {
            "r": 64,
            "lora_alpha": 128,
            "target_modules": [
                "q_proj",
                "k_proj",
                "v_proj",
                "o_proj",
                "gate_proj",
                "up_proj",
                "down_proj",
            ],
            "lora_dropout": 0.0,
            "description": "Maximum capacity for complex tasks",
        },
    }

    return configs.get(scenario)


# Display configurations
scenarios = [
    "classification_small_data",
    "instruction_tuning",
    "domain_adaptation",
    "code_generation",
]
Out[23]:
Console
Scenario: classification_small_data
  Description: Conservative config for limited data
  Rank: 8, Alpha: 8 (ratio: 1.0)
  Target: q_proj, v_proj
  Dropout: 0.1

Scenario: instruction_tuning
  Description: Balanced config for instruction following
  Rank: 16, Alpha: 32 (ratio: 2.0)
  Target: q_proj, k_proj, v_proj, o_proj
  Dropout: 0.05

Scenario: domain_adaptation
  Description: High-capacity config for domain shift
  Rank: 32, Alpha: 64 (ratio: 2.0)
  Target: q_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  Dropout: 0.05

Scenario: code_generation
  Description: Maximum capacity for complex tasks
  Rank: 64, Alpha: 128 (ratio: 2.0)
  Target: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  Dropout: 0.0
Out[24]:
Visualization
Bar chart of rank values across four scenarios, ascending from 8 to 64.
Rank values across four LoRA configuration scenarios, increasing from 8 for classification (small data) to 64 for code generation. The progression reflects increasing task complexity and the corresponding need for more representational capacity.
Bar chart of alpha/rank ratios across four scenarios, all either 1.0 or 2.0.
Alpha-to-rank ratio for four LoRA configuration scenarios. All scenarios use a ratio of either 1.0 or 2.0, with the higher ratio (2x) applied to scenarios requiring stronger adaptation relative to the pre-trained model's behavior.
Bar chart of target module counts across four scenarios, from 2 to 7.
Number of target modules for four LoRA configuration scenarios. Classification uses only 2 (Q, V), while instruction tuning uses 4 (all attention), and more complex scenarios extend to 6 or 7 modules including FFN projections.

The recipes reveal a coherent progression: as task complexity increases and the required behavioral change from the pre-trained model grows larger, both rank and the number of target modules increase. The alpha-to-rank ratio stays at either 1.0 or 2.0. This reflects the practical convention of choosing alpha as a simple multiple of rank rather than tuning it independently. Notice that dropout is higher for the low-data scenario (0.1) and absent for code generation (0.0). This reflects the regularization trade-off: with abundant data, dropout provides little benefit and slightly slows training.

A Complete Training Example

Here is how these hyperparameters come together in practice. This shows the configuration structure you would use with the PEFT library.

In[25]:
Code
from peft import LoraConfig, TaskType

# Recommended configuration for general instruction tuning
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",  # Attention
    ],
    lora_dropout=0.05,
    bias="none",  # Don't train biases
    task_type=TaskType.CAUSAL_LM,
)
Out[26]:
Console
LoRA Configuration Summary:
  Rank: 16
  Alpha: 32
  Effective scaling: 2.0
  Target modules: {'k_proj', 'o_proj', 'q_proj', 'v_proj'}
  Dropout: 0.05
  Bias training: none

The configuration targets all attention matrices with rank 16 and alpha 32, creating a scaling factor of 2. This amplified scaling helps the adapter make meaningful contributions despite its small parameter count. The modest dropout of 0.05 provides light regularization without significantly slowing convergence. This setup balances capacity with stability, which makes it a reliable starting point for instruction tuning tasks on modern decoder-only architectures.