PEFT Comparison › Key Parameters

Understanding which parameters control each method and how they interact is needed for effective hyperparameter tuning. The key setup classes and parameters for PEFT methods are:

LoraConfig controls the following key hyperparameters:

• r: Rank of the update matrices. Lower rank means fewer parameters and less memory. Common values are 4, 8, 16, 32, and 64. Starting with 16 is generally safe.
• lora_alpha: Scaling factor for updates. The effective weight scaling is $\alpha/r$. Setting $\alpha = 2r$ is a common default that keeps the effective learning rate stable across different rank settings.
• target_modules: List of module names (for example, "query", "value") to apply LoRA to. Including FFN modules alongside attention modules increases capacity at the cost of more parameters.
• lora_dropout: Dropout probability for LoRA layers, which reduces overfitting. Values of 0.05-0.1 work well for most fine-tuning tasks.

PrefixTuningConfig and PromptTuningConfig share similar key parameters:

• num_virtual_tokens: Number of learnable tokens to prepend. More tokens give more capacity but increase memory and context consumption. Values of 10-100 are typical.
• prompt_tuning_init: Initialization method for soft prompts. Random initialization is simplest; text-based initialization (starting from the embeddings of a manually crafted prompt) can accelerate convergence when a reasonable text prompt is available.