PEFT Comparison: Choosing the Right Fine-Tuning Method

Throughout this part, we've explored six distinct approaches to parameter-efficient fine-tuning: LoRA, QLoRA, Adapters, IA³, Prefix Tuning, and Prompt Tuning. Each method introduces a different inductive bias about where task-specific knowledge should reside in a model, and each shows a different theory about the minimum amount of change required to teach a pre-trained model something new. The diversity of approaches is not accidental. It shows real disagreement in the research community about whether adaptation should happen in the weights, in the activations, in the attention patterns, or somewhere in between.

Think of it this way: if a world-class chef were asked to modify their signature dish for a new dietary requirement, they might take different approaches. They could update the recipe itself (full fine-tuning). They could add a specialized sauce that compensates for what's removed (adapters). They could adjust the seasoning ratios without changing any ingredients (IA³). They could learn to describe the dish differently to diners (prompt tuning). Each approach changes what the diner experiences, but differs in the effort required, how easily it can be reversed, and whether it affects other dishes the chef must still prepare. PEFT methods work the same way: the six techniques we've studied stand for fundamentally different philosophies about how adaptation should happen, and the right choice depends heavily on what kind of "dish" you are preparing and for which "diner".

This chapter synthesizes what we've learned into a practical comparison framework. We'll examine each method along multiple dimensions: parameter efficiency, task performance, computational overhead, and implementation complexity. Rather than treating this as a simple ranking exercise, we'll analyze the tradeoffs that make each method the right choice for particular scenarios. Understanding why each method excels in some settings and struggles in others is more useful than any single ranked list, because your situation is unlikely to match precisely the benchmark conditions used in published comparisons.

The key insight organizing this chapter is that no method dominates all dimensions simultaneously. A method that trains fewer parameters will generally reach lower peak performance but fit on smaller hardware. A method that adds no inference overhead will constrain what it can learn at training time. A method that reaches high performance will require more data and compute to train. Every choice is a tradeoff, and the goal is to understand those tradeoffs deeply enough to make the right choice for your specific problem.

By the end, you'll have concrete guidelines for selecting the right PEFT method for your use case, anchored in the quantitative analysis we'll develop throughout the chapter. We'll also discuss when to abandon PEFT altogether and use full fine-tuning, because understanding the limits of efficiency is just as important as understanding its benefits.

PEFT methods adapt large models by updating only a small fraction of parameters. To make a useful comparison, we need concrete metrics rather than qualitative descriptions. Examining the parameter counts reveals how methods differ in memory and storage requirements, and these numbers connect directly to the hardware constraints you'll face in practice.

The central question is not just "how few parameters can a method use?" but "how efficiently do those parameters capture the necessary adaptation?" A method that uses 1% of parameters but reaches only 80% of full fine-tuning performance is less parameter-efficient in a practical sense than a method that uses 0.3% of parameters and reaches 97% performance. Throughout this section, keep in mind that parameter count is a proxy for what we care about: the combination of hardware requirements, storage costs, and training time that determines whether a method is practical for your situation.

We will work through the parameter calculations for each method applied to a common reference model, establishing a shared baseline that allows direct comparison. The numbers we derive here are not just theoretical exercises. They explain the real-world constraints you encounter when choosing between a 16GB GPU and a 48GB GPU, or between running five experiments in parallel versus two.

Let's establish concrete numbers for a typical 7B parameter model like LLaMA-2-7B with hidden dimension $d = 4096$, 32 layers, and 32 attention heads. Calculating the parameters for each method shows the tradeoffs and suitability for different hardware constraints.

LoRA approximates weight updates by decomposing them into two low-rank matrices, $A$ and $B$, such that $\Delta W = BA$. This design relies on the hypothesis that weight updates have a low "intrinsic rank," meaning the necessary adaptations exist in a low-dimensional subspace. The key insight motivating this approach is that while the original weight matrices are enormous, the changes needed to adapt a model to a new task may be far simpler in structure. If we can capture these changes using matrices with reduced dimensionality, we reach large parameter savings without sacrificing the model's ability to learn the task. For a layer with hidden dimension $d$ and rank $r$, the number of parameters is $d \times r + r \times d = 2dr$. When applied to all four attention projections ($W_Q, W_K, W_V, W_O$), we sum across all layers to get the total trainable parameter count.

The total number of trainable parameters for LoRA is calculated by summing the parameters of the low-rank matrices $A$ and $B$ across all target modules and layers. This calculation reveals how the rank parameter $r$ is a direct control knob for the tradeoff between model capacity and parameter count:

where:

• $4$: the number of target modules per layer (Query, Key, Value, Output)
• $2 \times d \times r$: parameters for matrices $A$ and $B$ in one module (since each has size $d \times r$ or $r \times d$)
• $d$: hidden dimension ($4096$)
• $r$: rank of the low-rank adapters ($16$)
• $L$: number of layers ($32$)

For our LLaMA-2-7B setup, this yields:

This result is striking: we can adapt a 7 billion parameter model by training only 16.8 million parameters, a reduction of over 400 times. The rank of 16 is deliberately chosen to be small enough to give substantial parameter savings while still giving sufficient capacity for most adaptation tasks. If the task proves to be complex, increasing the rank to 32 or 64 gives more capacity while still training only a small fraction of the model's parameters.

Adapters insert bottleneck modules, consisting of a down-projection and an up-projection, after both the attention and feed-forward (FFN) layers. This bottleneck architecture compresses inputs into a lower-dimensional representation before expanding them back, reducing parameter count while letting non-linear adaptation. The basic idea behind adapters is that task-specific information can be "filtered" through a narrow bottleneck, forcing the model to learn only the most needed transformations. This compression-expansion pattern also introduces non-linearity through activation functions, giving adapters additional representational power compared to linear methods like LoRA. With bottleneck dimension $d_{\text{bottleneck}} = 64$, we calculate the parameters for one adapter:

We calculate the parameter count by summing the weights of the down-projection and up-projection matrices for each adapter. The symmetry of this calculation, with equal contributions from each projection direction, shows the balanced nature of the bottleneck design:

where:

• $2$: the number of adapters per layer (one inserted after Attention, one after FFN)
• $d$: the hidden dimension of the model ($4096$)
• $d_{\text{bottleneck}}$: the size of the bottleneck dimension ($64$)
• $d \times d_{\text{bottleneck}}$: the number of parameters in the down-projection matrix
• $d_{\text{bottleneck}} \times d$: the number of parameters in the up-projection matrix
• $L$: the number of layers in the model ($32$)

Plugging in the values:

Adapters require roughly twice as many parameters as LoRA with rank 16. This increased parameter count buys additional expressiveness through the non-linear activation function between the down and up projections, but comes at the cost of both storage and, as we'll see later, inference overhead. The bottleneck dimension of 64 was chosen here to be illustrative; in practice, practitioners often tune this value just as they would tune LoRA's rank.

Prefix Tuning prepends learnable vectors to the keys and values at every layer. These virtual tokens effectively "steer" the attention mechanism at every depth, guiding the model's internal processing without modifying any weights. Rather than modifying the model's weights directly, prefix tuning modifies the context that the model attends to at every layer. Optimizing key-value pairs at every layer influences attention flow without changing the transformation matrices. With prefix length $l = 20$, we can calculate the total parameter count.

The trainable parameters consist of the virtual token embeddings added to the Key and Value matrices at each layer. This formulation makes clear why prefix tuning affects only the attention mechanism: we're adding content for the model to attend to, not modifying how it processes that content:

where:

• $l$: the prefix length (number of virtual tokens, set to $20$)
• $d$: the hidden dimension ($4096$)
• $2$: a factor accounting for application to both Key and Value matrices
• $L$: the number of layers ($32$)

Substituting the values:

The parameter count for Prefix Tuning is lower than LoRA despite operating at every layer. This efficiency comes from the method's focused scope: it only influences the attention mechanism through the key-value channel, leaving all other computations unchanged. However, this scope also explains one of Prefix Tuning's key weaknesses, which we'll analyze when we examine task performance.

Prompt Tuning adds learnable embeddings only at the input layer. Unlike discrete text prompts, these continuous embeddings are optimized directly via backpropagation to find the most effective task trigger. This approach is perhaps the simplest possible form of model adaptation: we're in effect learning a better way to "ask" the model to perform our task. The continuous nature of these embeddings allows optimization to find representations that may have no natural language equivalent, potentially discovering more effective task specifications than any human-written prompt could give. With prompt length $p = 100$:

The parameters come solely from the learnable embeddings prepended to the input layer. This single-layer application is what gives Prompt Tuning its remarkable parameter efficiency, though it also limits the method's ability to influence deep model computations:

where:

• $p$: the prompt length (number of virtual tokens, set to $100$)
• $d$: the embedding dimension ($4096$)

Calculating the total:

With fewer than half a million parameters, Prompt Tuning reaches the smallest footprint of any method we've examined. This extreme efficiency makes it ideal for scenarios where storage or memory is at an absolute premium, though as we'll see in the performance analysis, this efficiency comes with real tradeoffs.

IA³ introduces learned vectors that scale the activations element-wise. This acts like a feature equalizer, selectively amplifying or suppressing specific activation channels based on what the task requires. IA³ assumes the base model computes useful features and only needs to adjust their relative importance for different tasks. Rather than learning new transformations, IA³ learns which existing computations to emphasize and which to diminish. IA³ is effective because it uses pre-trained representations: the model has already learned to compute real features, and IA³ simply re-weights them for the task at hand. We calculate the parameter count by summing the scaling vectors for the Key, Value, and FFN intermediate activations:

where:

• $d_k, d_v$: the dimension of scaling vectors for attention keys and values ($4096$, matching the hidden dimension)
• $d_{\text{ff}}$: the dimension of the scaling vector for the FFN expansion layer ($11008$)
• $L$: the number of layers ($32$)

This results in a very small number of parameters:

The parameter count for IA³ is remarkably low, just over half a million for a 7B model. This efficiency stems from the element-wise nature of the scaling operation: we need only one scalar per activation dimension, rather than full transformation matrices. The tradeoff is that IA³ can only rescale existing activations, not compute fundamentally new features.

Parameter count alone doesn't tell the full story. During training, memory consumption includes model weights, optimizer states, gradients, and activations. Understanding the complete memory picture is needed for determining which methods will fit on your hardware. A method might train fewer parameters but still require substantial memory for other components of the training process.

Think of memory during training as having four distinct buckets that must all fit simultaneously into GPU VRAM. The first bucket holds the model weights themselves, which you must keep loaded at all times regardless of how many parameters you're updating. The second bucket holds the optimizer states, which for AdamW include two floating-point values per trainable parameter (the first and second moment estimates). The third bucket holds the gradients computed during backpropagation, which require storage for every trainable parameter. The fourth bucket holds the activations saved during the forward pass that are needed for gradient computation during the backward pass. PEFT methods reduce the second and third buckets to near zero for frozen parameters, but the first bucket remains unchanged.

For full fine-tuning with AdamW, memory is consumed by weights, optimizer states, gradients, and activations. Assuming mixed precision training, we can estimate the total memory requirement by analyzing the contribution of each component:

where:

• weights: memory for the model parameters (usually FP16), consuming 2 bytes per parameter
• optimizer: memory for optimizer states (momentum and variance in AdamW), consuming approximately 8 bytes per parameter in FP32
• gradients: memory for storing gradients during backpropagation, consuming approximately 4 bytes per parameter
• activations: memory for intermediate outputs needed for the backward pass, which scales with batch size and sequence length

Using a simplified estimation model:

where:

• $P$: the total number of model parameters
• $2P$: FP16 model weights ($2$ bytes per parameter)
• $4P$: optimizer states (simplified estimate for AdamW first and second moments)
• $4P$: gradients (FP32 or mixed precision)
• $2P$: activations (approximate average per layer)

For a 7B model, this works out to approximately $12 \times 7 \times 10^9 \approx 84$ GB of VRAM. Full fine-tuning is memory-intensive because requirements scale linearly with parameters, and each parameter needs storage for multiple values. No consumer GPU can come close to fitting these requirements.

PEFT methods drastically reduce memory by freezing the base model and training only the adapter parameters. The key insight is that frozen parameters need only their weights stored, not optimizer states or gradients. This is the most important memory saving PEFT gives: by freezing 99%+ of parameters, we eliminate the need to store optimizer states and gradients for those parameters, reducing the total memory requirement by a factor of 5-6x. With LoRA (rank 16):

For LoRA, we only need to store the optimizer states and gradients for the small set of trainable parameters, while the base model remains frozen in low precision. This separation between frozen and trainable parameters is the basic mechanism through which all PEFT methods reach their memory efficiency:

where:

• $P_{\text{frozen}}$: the number of frozen base model parameters ($7$ billion)
• $P_{\text{trainable}}$: the number of trainable LoRA parameters ($16.8$ million)
• $2$: bytes per frozen parameter (FP16 weights only, no optimizer state or gradient needed)
• $12$: the memory factor for trainable parameters (weights + optimizer + gradients + activations)

Applying this to our example:

This is why a 7B model that requires 80+ GB for full fine-tuning can be fine-tuned with LoRA on a single 24GB GPU. Memory savings make fine-tuning accessible; LoRA lets adaptation on hardware unsuitable for full fine-tuning. This shift in memory requirements lets you to adapt large models for your specific needs without expensive compute clusters or cloud GPU instances. The practical implication of this reduction cannot be overstated: it transformed large model fine-tuning from an activity requiring specialized infrastructure into something a graduate student with a gaming GPU can do.

Parameter efficiency means nothing if the method doesn't work. Understanding how different PEFT methods perform across task categories is needed for making informed selection decisions, because the performance profile of a method determines whether it is merely a memory-saving technique or a real replace full fine-tuning.

The performance picture is more fine-grained than a simple ranking would suggest. Methods that train more parameters tend to perform better in aggregate, but task type, data quantity, and model size all modulate this relationship. A method that performs poorly on generation tasks may excel at classification. A method that struggles with limited data may match full fine-tuning when data is abundant. We will examine performance along three dimensions that together cover the most common practical use cases: natural language understanding, text generation, and instruction following.

NLU tasks like sentiment analysis, natural language inference, and question answering have been extensively benchmarked. Classification and extraction tasks are the most studied PEFT domains and give reliable comparative data. On the GLUE benchmark, the relative ordering tends to follow the parameter count ordering, but with smaller gaps than you might expect:

Full Fine-tuning ≥ LoRA ≥ Adapters ≥ IA³ > Prefix Tuning > Prompt Tuning

The key insight here is that even the weakest PEFT methods often reach respectable performance. The differences at the bottom of this ranking are real but small in absolute terms: Prompt Tuning on GLUE might score 3-5 points below full fine-tuning, which matters for state-of-the-art comparisons but may be entirely acceptable for your application if you value the extreme parameter efficiency. Methods that train more parameters (LoRA, Adapters) reach performance closer to full fine-tuning, while the ultra-efficient methods (Prompt Tuning, IA³) show somewhat larger gaps. On BERT-base with the standard GLUE tasks:

LoRA typically reaches 95-99% of full fine-tuning performance while training less than 1% of parameters. This efficiency ratio makes PEFT methods attractive for practical applications. You can often get nearly all the benefit of full fine-tuning at a tiny fraction of the cost. The performance gap between LoRA and full fine-tuning on these NLU tasks is often within the noise of hyperparameter selection, suggesting that for many classification tasks, LoRA is a near-optimal tradeoff between efficiency and capability. IA³'s strong showing on NLU tasks (close to Adapters despite far fewer parameters) reflects the fact that classification benefits particularly from re-weighting existing features, which is precisely what IA³ does.

Generation tasks show different patterns from NLU. Summarization and translation are examples, as is dialogue generation. The quality of generated text depends heavily on how well the method can modify the model's output distribution across an entire sequence of predictions. Generation tasks are sensitive to adaptation quality because they require sequential predictions where small errors compound: a slight degradation in probability calibration per token becomes magnified across a 100-token output.

The architecture of each PEFT method determines how much influence it can exert over the generation process. Methods that modify weights throughout the network (LoRA, Adapters) have broad influence over both encoding and generation. Methods that operate only on attention (Prefix Tuning) miss the feed-forward computations that are important for generating specific vocabulary items. Methods that operate at a single layer (Prompt Tuning) must propagate their influence through the entire network indirectly.

Prefix Tuning shows a larger gap on generation tasks. This makes intuitive sense: as we discussed in the Prefix Tuning chapter, the method modifies attention patterns through prefix keys and values but doesn't directly alter how the model generates output tokens. The feed-forward networks, which play a useful role in transforming attended information into next-token predictions, remain entirely unchanged by prefix tuning. LoRA, by contrast, can modify both the attention mechanism and the FFN layers when configured to do so, giving it more direct control over the generation process. This architectural difference explains much of the performance gap you see on generation tasks, and it should influence your decision if you're working on summarization, translation, or any other output-heavy task.

The most practical application of PEFT today is adapting base models to follow instructions. This presents a unique challenge: the model must learn both the format of instruction-response pairs and diverse task knowledge simultaneously. Instruction following requires the model to understand when it should generate output, what style that output should take, and how to appropriately respond to the large diversity of requests. This combined learning objective places significant demands on the adaptation method's capacity, which is why this benchmark is particularly revealing about the practical utility of each approach.

The distinction between instruction following and standard NLG is important. In instruction following, the model must recognize the intent behind arbitrary requests and generate appropriate responses. Language fluency alone is insufficient; instruction following requires a form of meta-understanding that goes beyond pattern matching. Methods that can modify both attention patterns and FFN computations have an advantage because they can learn both what to attend to (via attention modification) and how to formulate responses (via FFN modification).

On instruction-following benchmarks, LoRA-tuned models often match fully fine-tuned versions:

QLoRA's strong performance here is remarkable: it reaches near-LoRA quality while letting fine-tuning of 7B models on consumer GPUs with just 24GB of memory. The combination of 4-bit quantization for the base model weights and full-precision LoRA adapters proves to be an excellent practical compromise. The quantization introduces a small but measurable noise floor that limits QLoRA to slightly below standard LoRA, but the gap is small enough that the memory savings typically justify the tradeoff. Prefix Tuning's lower scores here reflect its inability to modify FFN computations, which appear to be particularly important for the style and format aspects of instruction following. We'll explore instruction tuning in depth in the next part.

When training data is scarce (hundreds rather than thousands of examples), the relative performance of methods shifts materially. Data efficiency becomes a necessary concern when labeled examples are expensive to obtain, when working with specialized domains, or when rapidly prototyping new applications. Understanding how each method behaves in low-data regimes helps inform decisions about whether to invest in data collection or accept the performance tradeoffs of a more efficient method.

The intuition behind data efficiency differences comes down to the parameter budget. Methods with more trainable parameters need more data to prevent overfitting: if you have 16 million parameters and only 100 examples, the network can trivially memorize the training set without learning generalizable patterns. Methods with fewer trainable parameters are inherently more constrained, and this constraint acts as a form of regularization that can help in low-data settings.

The pattern is clear: methods that train more parameters (LoRA, Adapters) are more data-hungry but reach higher peak performance. Prompt Tuning struggles with limited data because it must learn rich representations from scratch in a very low-dimensional space. The soft prompt embeddings start as random vectors with no task-needed structure, and with only a few hundred examples, there simply isn't enough signal to learn effective representations. IA³, while also parameter-efficient, fares better in low-data settings because its scaling approach builds on the model's existing feature representations rather than learning new ones from scratch. This is a important practical insight: if you're working in a low-data regime, your choice of PEFT method matters more than in a data-rich setting. IA³ or LoRA with a small rank (r=4 or r=8) are often better choices than either Prompt Tuning or high-rank LoRA when you have only hundreds of examples.

Beyond memory, PEFT methods differ in their computational costs during training and inference. These differences can materially impact both development iteration speed and production serving costs. A complete understanding of PEFT tradeoffs requires examining parameter counts, performance, and the time and compute required to train and deploy adapted models.

The computational picture has two distinct phases: training time and inference time. These phases have very different cost structures. During training, you typically run it once (or a few times with different hyperparameters), so the absolute training time matters less than during inference, which runs continuously in production. A method that takes 50% longer to train but adds zero inference overhead may be far more economical overall than a method that trains quickly but adds 15% latency to every prediction.

Training throughput depends on several factors that interact in complex ways. Understanding these interactions helps explain why methods with fewer trainable parameters don't always train proportionally faster.

The factors that determine training speed are:

1. Forward pass overhead: Added computation from adapter layers or modified attention. Methods that introduce new layers (Adapters) or extend sequence length (Prefix Tuning) increase forward pass time, even when those additions are small in absolute terms.
2. Backward pass scope: How many parameters receive gradients. Methods with fewer trainable parameters compute fewer gradients and update fewer optimizer states, reducing backward pass time materially.
3. Memory access patterns: Sequential access (Adapters) vs. broadcasted scaling (IA³). Memory-efficient operations can give substantial speedups even when parameter counts are similar. IA³'s element-wise multiplication is particularly cache-friendly compared to the matrix multiplications required by LoRA adapters.

Throughput improves primarily from reduced gradient computation. Prompt Tuning is fastest because gradients flow through only 100 embedding vectors, making the backward pass nearly instantaneous relative to full fine-tuning. The backward pass is faster because most parameters are frozen and require no gradient computation. These speed improvements, while less large than the memory savings, materially accelerate the iterative development cycle when you need to run many experiments to identify the right hyperparameters.

A necessary distinction separates PEFT methods at inference time: some add persistent overhead while others are "free" after training. This distinction has major implications for production deployments, where inference costs often dominate the total cost of ownership for ML systems. A model that serves millions of requests per day at 1.15x the base latency costs 15% more in compute than the base model, which at scale translates to substantial ongoing infrastructure expenses.

The key property that determines inference impact is whether the method's adaptations can be merged into the base model weights, eliminating all traces of the adaptation at serving time:

Zero inference overhead (mergeable):

• LoRA: Weights can be merged ($W + BA$) after training. The low-rank decomposition is purely a training-time construct; once the adapter is trained, we can precompute the full weight update and fold it into the base model weights. After merging, the serving model is identical in structure to the original.
• IA³: Scaling vectors can be absorbed into the preceding linear transformation's weights through a simple element-wise multiplication of the weight matrix columns. The result is a slightly modified weight matrix with identical inference behavior.
• Prompt Tuning: Only adds constant prefix tokens, which have minimal impact on inference time (typically less than 5%) since they are constant vectors that add a small fixed overhead per sequence.

Persistent inference overhead:

• Adapters: Must execute additional layers at every forward pass. The bottleneck computations cannot be removed because they involve non-linear activations that cannot be merged with surrounding linear operations. Every request pays the computational cost of the adapter's down-projection, non-linearity, and up-projection.
• Prefix Tuning: Increases the effective sequence length by the prefix size at every layer. The prefix tokens add to the key-value cache and increase attention computation proportional to the prefix length, which can be significant for long-context applications.

For production deployments where inference cost dominates, LoRA's ability to merge weights is a significant advantage. You get the benefits of task-specific adaptation with identical serving costs to the base model. This property makes LoRA the preferred choice for applications where models must serve millions of requests, as even small percentage increases in latency translate to substantial infrastructure costs at scale. The decision between Adapters and LoRA for production systems often comes down to this single property: if you need dynamic task switching at inference time (loading different adapters for different requests), Adapters' architecture may justify the overhead; if you serve a single specialized model, LoRA's mergeability makes it the clear winner.

Different PEFT methods suit different scenarios. Mapping methods to use cases based on the technical properties we've analyzed gives actionable guidance that accounts for your specific constraints and requirements. The goal is not to give you a single ranked list but to explain the reasoning behind each recommendation deeply enough that you can adapt the guidance to situations not covered here. No single method is universally best; the right choice depends on your particular combination of performance needs, resource constraints, and deployment requirements.

Think of this analysis as a decision framework: you enter with a set of constraints (hardware, data volume, performance target, serving requirements) and exit with a justified recommendation. The constraints interact, and understanding those interactions is what separates principled method selection from guesswork.

LoRA is the default choice for most scenarios and excels when performance and practicality must be balanced. It consistently reaches the smallest gap to full fine-tuning across diverse task types and model scales, and its weight-merging property eliminates all inference overhead after training. For general fine-tuning with standard hardware and a reasonable amount of task data, starting with LoRA is almost always the right call.

More specifically, LoRA is the right choice when:

• You need maximum performance: LoRA consistently reaches the smallest gap to full fine-tuning across diverse task types and model scales, which makes it the safest default when performance is the primary concern.
• Inference cost matters: Merged weights mean zero serving overhead, making LoRA ideal for high-traffic production deployments where you need a single specialized model serving all requests.
• You are fine-tuning on GPUs with 24-48GB VRAM: Standard LoRA (r=16-64) fits comfortably on professional-grade consumer hardware, including NVIDIA RTX 4090 and A5000 class GPUs.
• You want to stack multiple adapters: Different LoRA weights can be loaded dynamically for multi-task serving, letting efficient model customization without maintaining separate model copies for each task.

Avoid LoRA when you have fewer than 100 training examples (consider few-shot prompting instead, since LoRA will likely overfit) or when memory is severely constrained even at low ranks (consider QLoRA or Prompt Tuning).

QLoRA extends LoRA's applicability to memory-constrained settings by combining 4-bit quantization of base model weights with standard LoRA adapters. The key insight is that quantization reduces the base model footprint from 2 bytes per parameter (FP16) to approximately 0.5 bytes per parameter (4-bit NF4), while the LoRA adapters themselves remain in full precision to preserve training quality. This combination reduces the memory required to fine-tune a 7B model from roughly 14GB to approximately 8GB, letting adaptation on widely available consumer GPUs with 8-12GB of VRAM.

QLoRA is the right choice when:

• Consumer hardware with limited VRAM: Fine-tune 7B models on 24GB GPUs, 13B on 48GB, making large model adaptation accessible without enterprise-grade hardware.
• Large models on limited budgets: Access to 70B-scale models without A100 clusters, democratizing research on frontier-scale models.
• Research and experimentation: Lower barrier to entry for exploring large models, letting faster iteration during the exploratory phase of projects where you're still deciding which model scale to use.

Avoid QLoRA when training speed is necessary (quantization adds roughly 10-20% overhead due to the need to dequantize weights during computation) or when you have sufficient memory for standard LoRA (standard LoRA trains faster and avoids any potential quality degradation from quantization).

Adapters are well-suited for scenarios that benefit from their modular, plug-and-play architecture. Unlike LoRA, whose adapters are designed to be merged and forgotten, Adapters are designed to be retained and swapped at inference time. This architectural philosophy makes them the right choice for systems that need to serve multiple tasks from a single base model by dynamically loading different adapter modules.

Adapters excel when:

• Multi-task learning: Each task gets its own adapter module, separating task-specific knowledge. You can add new tasks without modifying any existing adapters, making the system easily extensible.
• Modular architectures: Adapters compose naturally (task adapter stacked on top of language adapter), letting hierarchical organization of model capabilities. This is particularly useful in multilingual settings where you might combine a language-specific adapter with a task-specific adapter.
• Interpretability research: Adapter outputs can be probed to understand task-specific processing, as the discrete adapter boundaries give natural intervention points for understanding what the model has learned.

Avoid Adapters when inference latency is necessary (adapters add permanent overhead that cannot be eliminated through weight merging) or when maximum parameter efficiency is required (adapters train more parameters than LoRA at comparable performance levels).

Prefix Tuning is a good fit for controllable generation. It inserts learned prefix vectors into the key-value matrices at every layer, changing attention at each depth without modifying model weights. Use it when the goal is to steer output toward a specific style, domain, or format.

Prefix Tuning is the right choice when:

• NLG tasks where style control matters: Especially controllable generation where prefix vectors can encode style and tone attributes without modifying the core generation mechanism. The model can be conditioned to generate formal vs. informal text, technical vs. accessible explanations, or any other stylistic dimension.
• Interpretability studies: Prefix attention patterns reveal what the model "looks for," as you can analyze which positions in the prefix receive attention for different inputs and what semantic content those positions appear to encode.
• Moderate data regimes: Better data efficiency than Prompt Tuning due to per-layer parameterization, while still more parameter-efficient than LoRA or Adapters.

Avoid Prefix Tuning when working with very long contexts (prefix reduces effective context length by the prefix size at every layer, potentially cutting into your input budget) or when tasks require modifying feed-forward computation (prefix only affects attention, leaving FFN layers entirely unchanged).

Prompt Tuning is the simplest PEFT method and fits scenarios where extreme efficiency is the primary constraint. With fewer than 0.5 million trainable parameters for a 7B model, it allows thousands of task-specific adaptations to fit in the memory that would otherwise hold a single LoRA adapter. This extreme compactness makes it well-suited for large-scale multi-tenancy scenarios where many users each have their own personalized adaptation.

Prompt Tuning is the right choice when:

• Extremely large models accessible only through APIs: When you can access a model only for inference (not training), or when even inference-only deployment of adapter weights is challenging due to infrastructure constraints.
• Maximum parameter efficiency is the primary constraint: Less than 0.01% trainable parameters lets storage of thousands of task-specific adaptations, making personalized model adaptation economically feasible at large scale.
• Large-scale multi-tenancy: If you need to serve millions of users each with a slightly different model behavior, soft prompts are cheap enough to store and fast enough to apply that personalization becomes practical.

Avoid Prompt Tuning when training data is limited (Prompt Tuning is data-hungry, requiring thousands of examples to reach good performance), when tasks require significant behavioral change from the base model (the limited parameter budget constrains adaptation capacity), or when you need interpretable adaptations (soft prompts are not human-readable).

IA³ offers a unique balance between efficiency and capability through its activation scaling approach. By learning scaling vectors that re-weight activations rather than learning new transformations, IA³ reaches better performance per parameter than Prompt Tuning in most settings, while training fewer parameters than LoRA. Its particularly strong showing in few-shot settings makes it useful when data is scarce but you still need more than Prompt Tuning can deliver.

IA³ excels when:

• Few-shot settings: Better data efficiency than Prompt Tuning because it builds on existing representations rather than learning new ones from scratch. The scaling approach amplifies signals that already exist in the model, which requires less data to converge than learning new transformations.
• Minimal overhead: Scales apply through element-wise multiplication, which is computationally trivial and adds no inference latency when folded into weights. You get more capability than Prompt Tuning at in effect the same serving cost.
• Quick experiments and ablations: Fastest to train among methods that modify attention, letting rapid prototyping and ablation studies when you need to quickly explore whether fine-tuning can solve your problem.

Avoid IA³ when maximum performance is required (typically 1-2% below LoRA on most benchmarks) or when the task requires learning entirely new capabilities that the base model has no signal for (limited expressiveness constrains what the model can learn from the existing activation patterns).

The guidelines above describe ideal scenarios, but real projects involve messier combinations of constraints. This section synthesizes the analysis into concrete, actionable protocols for the most common situations you'll encounter.

If you are fine-tuning LLMs for downstream tasks and don't have strong constraints pulling you toward a specialized method, the following protocol will serve you well in the large majority of cases:

1. Start with LoRA ($r=16$, $\alpha=32$) applied to all attention projections. This is the baseline that the research community has converged on, and it will give you a strong result to compare against.
2. Use 4-bit QLoRA if you're GPU-memory constrained. QLoRA loses at most 1-2% performance compared to full-precision LoRA while cutting memory requirements approximately in half.
3. Increase rank to 32-64 if performance is insufficient after initial experiments. Higher rank gives more adaptation capacity, and the additional memory cost is manageable on professional hardware.
4. Add FFN adaptation (LoRA on up/down projections) for tasks requiring significant behavioral change, such as instruction following or domain adaptation where the model's output style needs substantial modification.

When serving fine-tuned models in production, inference efficiency becomes the dominant concern. The recommendations differ from training-time considerations because production costs accumulate with every request:

1. Merge LoRA weights into the base model for zero inference overhead. The merged model is indistinguishable from a natively trained model in terms of serving infrastructure, and you lose nothing by merging.
2. Avoid Adapters unless you need runtime adapter switching. The 15% inference overhead of Adapters is only justified if you need to swap task-specific adapters at inference time without reloading the model.
3. For multi-tenant serving: keep LoRA weights separate and apply at runtime with specialized serving infrastructure. This makes possible personalized model behavior without requiring separate model instances for each user, though it does prevent the zero-overhead benefit of merging.

When exploring new tasks or model capabilities, development velocity matters more than peak performance. The goal during research is to iterate quickly and answer questions about feasibility before committing to expensive full experiments:

1. Start with Prompt Tuning for fastest iteration. If Prompt Tuning can solve your task, the overhead of training is minimal and you'll get answers quickly. Treat this as a feasibility check.
2. Move to IA³ if Prompt Tuning underperforms. IA³ gives substantially more capacity than Prompt Tuning with minimal additional overhead, and its few-shot efficiency makes it particularly well-suited for early-stage research with limited labeled data.
3. Use LoRA for final experiments requiring best performance. Once you've confirmed the task is solvable and have a reasonable understanding of the data requirements, switch to LoRA for the experiments you'll report and deploy.

When working with very limited compute, the goal is to get any fine-tuning to work at all rather than optimize for the last percentage point of performance:

1. QLoRA with $r=8$ gives excellent efficiency. The combination of 4-bit quantization and very low rank means you can fine-tune large models on hardware that would otherwise be completely inadequate.
2. Gradient checkpointing trades compute for memory. By recomputing activations during the backward pass rather than storing them, gradient checkpointing can cut activation memory by 60-80% at the cost of approximately 30% more compute. This is often worth the tradeoff when memory is the binding constraint.
3. Consider API-based fine-tuning if available for your model. Services like OpenAI's fine-tuning API handle the compute and infrastructure, requiring only that you give your training data.

Let's implement a framework for comparing PEFT methods on a real task. This hands-on comparison will reinforce the theoretical analysis with concrete measurements, letting you to see the parameter counts and training dynamics we've discussed in action on real models.

Prompt Tuning and IA³ offer the highest training throughput, requiring gradients for only a small number of parameters. LoRA and Prefix Tuning also give speedups over full fine-tuning, though the exact gain depends on the rank and number of adapted layers. These speed improvements, while less large than the memory savings, materially accelerate the iterative development cycle. Faster training means more experiments per day, faster hyperparameter searches, and quicker iteration toward optimal configurations. In practice, the ability to run more experiments often matters more than the raw performance of any single run.

An emerging research direction explores combining multiple PEFT methods for improved performance and flexibility. The intuition is that different methods modify different aspects of model behavior, and their complementary strengths might compound when combined thoughtfully.

Think of combining PEFT methods like layering different types of adjustments to a complex system. A musician recording a song might adjust the EQ (frequency balance), apply compression (dynamic range), and add reverb (spatial characteristics) simultaneously. Each adjustment targets a different dimension of the audio, and combining them reaches results that no single adjustment could produce alone. Similarly, combining LoRA's weight modifications with activation-level adjustments from IA³ targets different computational pathways simultaneously.

LoRA combined with Adapters is one natural combination: apply LoRA to attention layers and adapters after FFN layers. This captures both low-rank attention modifications and complex post-FFN transformations that might require the non-linearity of the bottleneck architecture. The tradeoff is that you inherit the inference overhead of adapters alongside the benefits of LoRA.

Prompt Tuning combined with LoRA offers another interesting combination: use soft prompts for task-level conditioning while LoRA handles fine-grained adaptation. This separation of concerns, where the prompt handles high-level task specification and LoRA handles low-level behavioral adjustment, is particularly useful for instruction-following scenarios with multiple task types.

Applying LoRA to both attention and FFN layers is the most common "combination" approach used in practice. It materially increases the parameter count compared to attention-only LoRA but gives much more complete adaptation capacity. For a 7B model with rank 16 applied to all seven module types listed above, the parameter count roughly doubles compared to attention-only LoRA, but remains well within the range that fits on consumer hardware with QLoRA.

Understanding which parameters control each method and how they interact is needed for effective hyperparameter tuning. The key setup classes and parameters for PEFT methods are:

LoraConfig controls the following key hyperparameters:

• r: Rank of the update matrices. Lower rank means fewer parameters and less memory. Common values are 4, 8, 16, 32, and 64. Starting with 16 is generally safe.
• lora_alpha: Scaling factor for updates. The effective weight scaling is $\alpha/r$. Setting $\alpha = 2r$ is a common default that keeps the effective learning rate stable across different rank settings.
• target_modules: List of module names (for example, "query", "value") to apply LoRA to. Including FFN modules alongside attention modules increases capacity at the cost of more parameters.
• lora_dropout: Dropout probability for LoRA layers, which reduces overfitting. Values of 0.05-0.1 work well for most fine-tuning tasks.

PrefixTuningConfig and PromptTuningConfig share similar key parameters:

• num_virtual_tokens: Number of learnable tokens to prepend. More tokens give more capacity but increase memory and context consumption. Values of 10-100 are typical.
• prompt_tuning_init: Initialization method for soft prompts. Random initialization is simplest; text-based initialization (starting from the embeddings of a manually crafted prompt) can accelerate convergence when a reasonable text prompt is available.

Despite their efficiency, PEFT methods have inherent limitations that should inform your choice about whether to use them at all. Understanding these limitations is as important as understanding their benefits, because deploying a PEFT method in a situation where it will fail creates more problems than it solves.

The most basic limitation is constrained adaptation capacity. By design, PEFT methods limit what the model can learn. For tasks requiring substantial behavioral change, such as learning a new language, acquiring domain expertise in a highly technical field, or fundamentally shifting the model's output format and style, full fine-tuning may be necessary. LoRA's low-rank constraint assumes task-specific modifications live in a low-dimensional subspace, which may not hold for all tasks. When you're trying to teach a model a new programming language it was never exposed to during pre-training, or when you're attempting to align a base model for safety properties, the limited parameter budget of PEFT methods may simply be insufficient to encode the necessary knowledge and behavioral patterns.

A second important limitation is sensitivity to hyperparameters. PEFT methods introduce new hyperparameters (rank, prefix length, bottleneck dimension) that require tuning. Poor choices can materially impact performance, sometimes more sharply than learning rate selection in full fine-tuning. The rank parameter in LoRA is particularly sensitive: a rank that is too low will leave performance on the table, while a rank that is too high may cause overfitting on small datasets. Finding the right rank often requires running multiple experiments, which partially offsets the efficiency benefits of PEFT if your compute budget for experimentation is limited.

Degradation at scale is a third limitation that becomes particularly needed as systems grow more complex. While PEFT methods work well for single-task adaptation, repeatedly applying multiple LoRA adapters or stacking adapters can lead to interference effects. When two adapters have been trained on conflicting objectives, combining them produces unpredictable behavior that may be worse than either adapter alone. Research on LoRA merging and multi-task adapter composition remains an active area with no fully satisfying solution.

PEFT methods also do not replace pre-training. They adapt existing capabilities; they don't create new ones. A model that doesn't encode knowledge about a topic from pre-training generally won't learn it from PEFT alone, regardless of how much task-specific data you give. This is why PEFT works better on tasks that are already related to the model's pre-training distribution than on tasks that require fundamentally new knowledge. If your task falls outside the model's pre-training domain, you may need either a different base model (one pre-trained on needed data) or full fine-tuning with domain-specific data at a scale that can modify the model's knowledge encoding.

When should you opt for full fine-tuning despite the efficiency benefits of PEFT?

• Maximum performance is non-negotiable: In high-stakes applications where even a 1-2% gap matters, full fine-tuning remains the gold standard for squeezing out the best possible performance.
• Sufficient compute is available and cost is secondary: If you have the resources and the time, full fine-tuning removes uncertainty about whether a PEFT method's constraints are limiting your results.
• The task requires deep behavioral change: Domain adaptation, safety fine-tuning, developing a new capability, or adapting a model to a radically new domain may exceed PEFT's adaptation budget.
• You're creating a foundation for further PEFT: Full fine-tuning a domain-specific base model (for example, training a biomedical language model from a general LLM) and then using PEFT for specific downstream tasks is a powerful two-stage strategy that combines the strengths of both approaches.

Consider a concrete scenario: you need to build a question-answering system for clinical medical records. The system will answer questions about patient history using text from medical notes. Let's walk through the method selection process systematically.

Step 1: Characterize the task. Medical QA is an information extraction task (NLU-dominant), not pure generation. The model needs to identify needed information in context and produce concise, accurate answers. It will also encounter domain-specific terminology not well-represented in general pre-training data.

Step 2: Inventory your constraints. Suppose you have access to a single NVIDIA A100 with 40GB VRAM, 50,000 labeled question-answer pairs from clinical records, and strict latency requirements of under 200ms per query in production. You cannot afford any inference overhead.

Step 3: Apply the decision framework. Memory is not a constraint (40GB A100 handles standard LoRA on 7B models easily), so QLoRA is not necessary. Inference latency is necessary (under 200ms with zero overhead), which eliminates Adapters. Data is plentiful (50,000 examples), so Prompt Tuning's data-hungry nature is less of a concern, but LoRA's superior performance is still preferred. The task requires domain adaptation (medical terminology), suggesting higher rank is warranted.

Step 4: Make the decision. Use LoRA with $r=32$ applied to all attention projections and the FFN layers, with $\alpha=64$. Merge the weights after training for zero inference overhead. The $r=32$ rank gives sufficient capacity for domain adaptation, while the FFN adaptation helps with the specialized vocabulary. After fine-tuning, merge the LoRA weights into the base model to reach the latency requirement.

Step 5: Plan the validation. Compare the LoRA model against a full fine-tuning baseline on a held-out test set. If the gap exceeds 3% on clinical QA accuracy, consider whether the additional compute of full fine-tuning justifies the performance gain. If the gap is under 2%, the LoRA approach is validated as production-ready.

This worked example illustrates how the abstract decision framework translates into concrete choices. The key is working through constraints in order (hardware, then deployment, then data, then performance) and understanding which constraints eliminate which methods.

This chapter synthesized our exploration of parameter-efficient fine-tuning methods into practical guidance for method selection. The core message is that every PEFT method involves tradeoffs, and understanding those tradeoffs deeply is what lets principled method selection.

Parameter efficiency varies by orders of magnitude across methods. Prompt Tuning and IA³ train fewer than 0.01% of parameters, while LoRA and Adapters train 0.2-0.5%. This translates directly to memory savings: a 7B model requiring 84GB for full fine-tuning can be adapted with LoRA using only 14GB. QLoRA pushes this further, letting adaptation on hardware with as little as 8GB by quantizing the base model weights to 4-bit precision.

Performance gaps are smaller than efficiency gains. LoRA typically reaches 95-99% of full fine-tuning performance on NLU and NLG tasks, including instruction-following tasks. The gap narrows further with larger models and more training data, suggesting that as models continue to scale, PEFT methods will become increasingly viable substitutes for full fine-tuning across an ever-broader range of tasks.

Method choice depends on your specific constraint profile. LoRA is the default recommendation for most scenarios due to its strong performance and zero inference overhead when weights are merged. QLoRA extends this to memory-constrained settings. Adapters suit multi-task modular architectures where runtime task switching is needed. Prefix and Prompt Tuning offer maximum efficiency at some performance cost, making them useful for research workflows and multi-tenancy systems.

Inference impact matters for production. LoRA and IA³ can be merged into base weights for identical serving costs. Adapters add permanent inference overhead but let runtime task switching. For high-traffic applications, the mergeability of LoRA is often the deciding factor.

The limitations of PEFT are real and should inform method selection. For tasks requiring deep behavioral change, for safety-necessary applications demanding maximum performance, or for domains poorly covered by pre-training data, full fine-tuning may be the appropriate choice even when PEFT is technically feasible.

As we move into Part XXXVI on instruction tuning, these PEFT methods, especially LoRA and QLoRA, will become needed tools. Instruction tuning turns base models into capable assistants, and doing so efficiently is often the difference between practical application and theoretical interest. The combination of strong base models with efficient fine-tuning methods has, more than any other development in recent years, made advanced NLP capabilities accessible to practitioners working outside of the largest technology organizations.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about comparing and selecting PEFT methods.