QLoRA › Practical Hyperparameter Choices

Successfully applying QLoRA requires making good decisions across several interdependent hyperparameters. The choices interact in non-obvious ways, and getting them right can mean the difference between a fine-tuned model that performs at par with full fine-tuning and one that underperforms despite the sophisticated underlying machinery. Unlike standard LoRA where the hyperparameter choices are relatively well-established, QLoRA introduces additional variables related to quantization that need careful consideration.

The most important choice is the LoRA rank. A common mistake is using the same rank you'd use for standard LoRA (often 4-16) without adjusting for QLoRA's specific properties. Because the quantized base model introduces a small but non-zero noise floor into every forward pass, the LoRA adapters are doing two things simultaneously: compensating for quantization noise and learning the actual task adaptation. Higher ranks give the adapters more capacity to handle both jobs. In practice, ranks of 16-64 tend to work well for QLoRA applied to 7-13B models. For tasks very similar to what the base model was trained on, lower ranks work well. For tasks requiring specialized knowledge or reasoning patterns, higher ranks provide better results.

The learning rate for QLoRA adapters typically needs to be set more carefully than for standard LoRA. The effective learning rate interacts with the quantization noise: if the learning rate is too high, the adapters can overfit to the noise rather than the signal, potentially degrading performance on out-of-distribution examples. Starting with a learning rate of $2 \times 10^{-4}$ and using a cosine learning rate schedule with a warmup period of 2-5% of total steps is a reasonable default. The warmup period is particularly important because the combination of BF16 adapter updates on top of NF4 base weights can create numerical instabilities in the first few steps if the learning rate starts too high.

The selection of which weight matrices to apply LoRA adapters to significantly affects both memory and quality. QLoRA is typically applied to the projection matrices in the self-attention mechanism, but extending it to the feed-forward network layers often improves results, especially for knowledge-intensive tasks.

The attention projection matrices (query, key, value, and output) are the default targets because they control information flow and contextualization, the mechanisms most relevant to instruction following and task adaptation. However, the FFN layers in a transformer contain a surprisingly large fraction of the factual knowledge stored during pretraining. The feedforward network operates as a kind of associative memory, where the first linear layer creates a high-dimensional intermediate representation and the second projects it back down. When fine-tuning for domain-specific tasks or tasks requiring specialized factual knowledge, adding LoRA adapters to the gate and up projections in the FFN can provide meaningful performance improvements.

A practical approach is to start with attention-only adapters, evaluate performance, and then add FFN adapters if performance is insufficient. The memory cost of adding FFN adapters is small because the adapter matrices themselves are low-rank. The specific layer names to target vary by model architecture. LLaMA-style models use names like gate_proj, up_proj, and down_proj for FFN layers. Other models use different conventions, so consulting the model card or source code is necessary when extending beyond the standard attention targets.

Gradient checkpointing is almost always necessary with QLoRA when fine-tuning models above 7B parameters. Without it, activations from all forward pass computations must be stored in memory simultaneously during the backward pass, which can easily exceed available GPU memory for long sequences or large batches. With gradient checkpointing enabled, only a subset of activations are stored, and the rest are recomputed during the backward pass. This reduces peak activation memory by roughly 60-70% at the cost of approximately 30-40% additional computation time.

The effective batch size matters significantly for training quality. Smaller effective batch sizes can lead to noisy gradient estimates that don't provide as clear a direction for optimization. For QLoRA fine-tuning, effective batch sizes of 16-64 work well for most instruction-following tasks. Because QLoRA is often memory-constrained, gradient accumulation is typically the way to achieve a larger effective batch size without exceeding GPU memory. A batch size of 2 with 8 gradient accumulation steps gives the same gradient estimate as a batch size of 16, but requires only 2 examples worth of activations in memory at any moment.