PEFT Comparison › Summary

This chapter synthesized our exploration of parameter-efficient fine-tuning methods into practical guidance for method selection. The core message is that every PEFT method involves tradeoffs, and understanding those tradeoffs deeply is what lets principled method selection.

Parameter efficiency varies by orders of magnitude across methods. Prompt Tuning and IA³ train fewer than 0.01% of parameters, while LoRA and Adapters train 0.2-0.5%. This translates directly to memory savings: a 7B model requiring 84GB for full fine-tuning can be adapted with LoRA using only 14GB. QLoRA pushes this further, letting adaptation on hardware with as little as 8GB by quantizing the base model weights to 4-bit precision.

Performance gaps are smaller than efficiency gains. LoRA typically reaches 95-99% of full fine-tuning performance on NLU and NLG tasks, including instruction-following tasks. The gap narrows further with larger models and more training data, suggesting that as models continue to scale, PEFT methods will become increasingly viable substitutes for full fine-tuning across an ever-broader range of tasks.

Method choice depends on your specific constraint profile. LoRA is the default recommendation for most scenarios due to its strong performance and zero inference overhead when weights are merged. QLoRA extends this to memory-constrained settings. Adapters suit multi-task modular architectures where runtime task switching is needed. Prefix and Prompt Tuning offer maximum efficiency at some performance cost, making them useful for research workflows and multi-tenancy systems.

Inference impact matters for production. LoRA and IA³ can be merged into base weights for identical serving costs. Adapters add permanent inference overhead but let runtime task switching. For high-traffic applications, the mergeability of LoRA is often the deciding factor.

The limitations of PEFT are real and should inform method selection. For tasks requiring deep behavioral change, for safety-necessary applications demanding maximum performance, or for domains poorly covered by pre-training data, full fine-tuning may be the appropriate choice even when PEFT is technically feasible.

As we move into Part XXXVI on instruction tuning, these PEFT methods, especially LoRA and QLoRA, will become needed tools. Instruction tuning turns base models into capable assistants, and doing so efficiently is often the difference between practical application and theoretical interest. The combination of strong base models with efficient fine-tuning methods has, more than any other development in recent years, made advanced NLP capabilities accessible to practitioners working outside of the largest technology organizations.