PEFT Comparison › Practical Recommendations

The guidelines above describe ideal scenarios, but real projects involve messier combinations of constraints. This section synthesizes the analysis into concrete, actionable protocols for the most common situations you'll encounter.

If you are fine-tuning LLMs for downstream tasks and don't have strong constraints pulling you toward a specialized method, the following protocol will serve you well in the large majority of cases:

1. Start with LoRA ($r=16$, $\alpha=32$) applied to all attention projections. This is the baseline that the research community has converged on, and it will give you a strong result to compare against.
2. Use 4-bit QLoRA if you're GPU-memory constrained. QLoRA loses at most 1-2% performance compared to full-precision LoRA while cutting memory requirements approximately in half.
3. Increase rank to 32-64 if performance is insufficient after initial experiments. Higher rank gives more adaptation capacity, and the additional memory cost is manageable on professional hardware.
4. Add FFN adaptation (LoRA on up/down projections) for tasks requiring significant behavioral change, such as instruction following or domain adaptation where the model's output style needs substantial modification.

When serving fine-tuned models in production, inference efficiency becomes the dominant concern. The recommendations differ from training-time considerations because production costs accumulate with every request:

1. Merge LoRA weights into the base model for zero inference overhead. The merged model is indistinguishable from a natively trained model in terms of serving infrastructure, and you lose nothing by merging.
2. Avoid Adapters unless you need runtime adapter switching. The 15% inference overhead of Adapters is only justified if you need to swap task-specific adapters at inference time without reloading the model.
3. For multi-tenant serving: keep LoRA weights separate and apply at runtime with specialized serving infrastructure. This makes possible personalized model behavior without requiring separate model instances for each user, though it does prevent the zero-overhead benefit of merging.

When exploring new tasks or model capabilities, development velocity matters more than peak performance. The goal during research is to iterate quickly and answer questions about feasibility before committing to expensive full experiments:

1. Start with Prompt Tuning for fastest iteration. If Prompt Tuning can solve your task, the overhead of training is minimal and you'll get answers quickly. Treat this as a feasibility check.
2. Move to IA³ if Prompt Tuning underperforms. IA³ gives substantially more capacity than Prompt Tuning with minimal additional overhead, and its few-shot efficiency makes it particularly well-suited for early-stage research with limited labeled data.
3. Use LoRA for final experiments requiring best performance. Once you've confirmed the task is solvable and have a reasonable understanding of the data requirements, switch to LoRA for the experiments you'll report and deploy.

When working with very limited compute, the goal is to get any fine-tuning to work at all rather than optimize for the last percentage point of performance:

1. QLoRA with $r=8$ gives excellent efficiency. The combination of 4-bit quantization and very low rank means you can fine-tune large models on hardware that would otherwise be completely inadequate.
2. Gradient checkpointing trades compute for memory. By recomputing activations during the backward pass rather than storing them, gradient checkpointing can cut activation memory by 60-80% at the cost of approximately 30% more compute. This is often worth the tradeoff when memory is the binding constraint.
3. Consider API-based fine-tuning if available for your model. Services like OpenAI's fine-tuning API handle the compute and infrastructure, requiring only that you give your training data.