PEFT Motivation › The PEFT Ecosystem

Multiple PEFT methods exist, each with different trade-offs between memory efficiency and performance, with different implementation complexity. Understanding the available techniques helps you choose the right approach for a given problem. The upcoming chapters cover these techniques in depth, but it is useful to have a high-level map before examining the details.

LoRA (Low-Rank Adaptation) adds trainable low-rank matrices to existing weight matrices. It is the most popular PEFT method, balancing simplicity and performance while remaining efficient. The core idea is to decompose the weight update $\Delta W$ into a product of two small matrices $B$ and $A$, where the rank $r$ of this decomposition is much smaller than the original weight dimension. We will explore LoRA's mathematics and implementation in the next three chapters.

QLoRA combines LoRA with aggressive quantization, enabling fine-tuning of 70B models on a single GPU by storing base weights in 4-bit precision while keeping the LoRA adapter weights in 16-bit. The quantization reduces the frozen weight storage from 130 GB to about 35 GB for LLaMA-70B, bringing it within reach of a single A100-40GB GPU. QLoRA represents the current practical frontier for large-model fine-tuning on constrained hardware.

Prefix tuning and prompt tuning add trainable continuous vectors to the input, effectively learning soft prompts that steer the model's behavior without touching any of its weights. These methods modify even fewer parameters than LoRA but may underperform on tasks that require structural changes to the model's representations rather than just steering its attention. Prompt tuning in particular has been shown to match full fine-tuning when the base model is sufficiently large (above roughly 10B parameters), supporting the hypothesis that larger models have richer internal structure that can be activated through small input modifications.

Adapter layers, introduced by Houlsby et al. in 2019, insert small bottleneck modules between transformer layers. This was one of the first PEFT methods and remains effective, though LoRA has largely superseded it in practice. The main advantage of the Houlsby adapter is interpretability: it is easy to identify which layers are being adapted and how much capacity is being used. The main disadvantage is inference overhead, since the adapter modules add latency to every forward pass, whereas LoRA can be mathematically merged with the original weights at inference time.