LoRA Hyperparameters › Key Parameters

The core parameters that govern LoRA configuration can be summarized concisely:

• Rank ($r$): Determines the capacity of the update matrices. Start with 8-16 for simple tasks, increasing to 32-64 for complex adaptations. Look for the knee in the performance-vs-rank curve.

• Alpha ($\alpha$): Controls the strength of the adaptation relative to the pre-trained weights. Often set equal to rank (1x scaling) or double the rank (2x scaling). Fixed alpha values of 16 or 32 across rank choices are also common.

• Target Modules: Determines which model layers are adapted. Attention projections (Q, V) are the standard starting point; FFN layers can be added for increased capacity or domain knowledge injection.

• Dropout: Provides regularization for the adapter. Typical values are 0.05-0.1, with higher values used for smaller datasets. Set to 0 for large, abundant datasets.

The next chapter introduces QLoRA, which combines LoRA with quantization to enable fine-tuning of large models on consumer hardware by storing the frozen model in reduced precision. The hyperparameter principles established here carry over directly to QLoRA, making this chapter's guidelines a foundation for the quantization-aware fine-tuning methods that follow.