PEFT Comparison › Performance Comparison Across Tasks

Parameter efficiency means nothing if the method doesn't work. Understanding how different PEFT methods perform across task categories is needed for making informed selection decisions, because the performance profile of a method determines whether it is merely a memory-saving technique or a real replace full fine-tuning.

The performance picture is more fine-grained than a simple ranking would suggest. Methods that train more parameters tend to perform better in aggregate, but task type, data quantity, and model size all modulate this relationship. A method that performs poorly on generation tasks may excel at classification. A method that struggles with limited data may match full fine-tuning when data is abundant. We will examine performance along three dimensions that together cover the most common practical use cases: natural language understanding, text generation, and instruction following.

NLU tasks like sentiment analysis, natural language inference, and question answering have been extensively benchmarked. Classification and extraction tasks are the most studied PEFT domains and give reliable comparative data. On the GLUE benchmark, the relative ordering tends to follow the parameter count ordering, but with smaller gaps than you might expect:

Full Fine-tuning ≥ LoRA ≥ Adapters ≥ IA³ > Prefix Tuning > Prompt Tuning

The key insight here is that even the weakest PEFT methods often reach respectable performance. The differences at the bottom of this ranking are real but small in absolute terms: Prompt Tuning on GLUE might score 3-5 points below full fine-tuning, which matters for state-of-the-art comparisons but may be entirely acceptable for your application if you value the extreme parameter efficiency. Methods that train more parameters (LoRA, Adapters) reach performance closer to full fine-tuning, while the ultra-efficient methods (Prompt Tuning, IA³) show somewhat larger gaps. On BERT-base with the standard GLUE tasks:

LoRA typically reaches 95-99% of full fine-tuning performance while training less than 1% of parameters. This efficiency ratio makes PEFT methods attractive for practical applications. You can often get nearly all the benefit of full fine-tuning at a tiny fraction of the cost. The performance gap between LoRA and full fine-tuning on these NLU tasks is often within the noise of hyperparameter selection, suggesting that for many classification tasks, LoRA is a near-optimal tradeoff between efficiency and capability. IA³'s strong showing on NLU tasks (close to Adapters despite far fewer parameters) reflects the fact that classification benefits particularly from re-weighting existing features, which is precisely what IA³ does.

Generation tasks show different patterns from NLU. Summarization and translation are examples, as is dialogue generation. The quality of generated text depends heavily on how well the method can modify the model's output distribution across an entire sequence of predictions. Generation tasks are sensitive to adaptation quality because they require sequential predictions where small errors compound: a slight degradation in probability calibration per token becomes magnified across a 100-token output.

The architecture of each PEFT method determines how much influence it can exert over the generation process. Methods that modify weights throughout the network (LoRA, Adapters) have broad influence over both encoding and generation. Methods that operate only on attention (Prefix Tuning) miss the feed-forward computations that are important for generating specific vocabulary items. Methods that operate at a single layer (Prompt Tuning) must propagate their influence through the entire network indirectly.

Prefix Tuning shows a larger gap on generation tasks. This makes intuitive sense: as we discussed in the Prefix Tuning chapter, the method modifies attention patterns through prefix keys and values but doesn't directly alter how the model generates output tokens. The feed-forward networks, which play a useful role in transforming attended information into next-token predictions, remain entirely unchanged by prefix tuning. LoRA, by contrast, can modify both the attention mechanism and the FFN layers when configured to do so, giving it more direct control over the generation process. This architectural difference explains much of the performance gap you see on generation tasks, and it should influence your decision if you're working on summarization, translation, or any other output-heavy task.

The most practical application of PEFT today is adapting base models to follow instructions. This presents a unique challenge: the model must learn both the format of instruction-response pairs and diverse task knowledge simultaneously. Instruction following requires the model to understand when it should generate output, what style that output should take, and how to appropriately respond to the large diversity of requests. This combined learning objective places significant demands on the adaptation method's capacity, which is why this benchmark is particularly revealing about the practical utility of each approach.

The distinction between instruction following and standard NLG is important. In instruction following, the model must recognize the intent behind arbitrary requests and generate appropriate responses. Language fluency alone is insufficient; instruction following requires a form of meta-understanding that goes beyond pattern matching. Methods that can modify both attention patterns and FFN computations have an advantage because they can learn both what to attend to (via attention modification) and how to formulate responses (via FFN modification).

On instruction-following benchmarks, LoRA-tuned models often match fully fine-tuned versions:

QLoRA's strong performance here is remarkable: it reaches near-LoRA quality while letting fine-tuning of 7B models on consumer GPUs with just 24GB of memory. The combination of 4-bit quantization for the base model weights and full-precision LoRA adapters proves to be an excellent practical compromise. The quantization introduces a small but measurable noise floor that limits QLoRA to slightly below standard LoRA, but the gap is small enough that the memory savings typically justify the tradeoff. Prefix Tuning's lower scores here reflect its inability to modify FFN computations, which appear to be particularly important for the style and format aspects of instruction following. We'll explore instruction tuning in depth in the next part.

When training data is scarce (hundreds rather than thousands of examples), the relative performance of methods shifts materially. Data efficiency becomes a necessary concern when labeled examples are expensive to obtain, when working with specialized domains, or when rapidly prototyping new applications. Understanding how each method behaves in low-data regimes helps inform decisions about whether to invest in data collection or accept the performance tradeoffs of a more efficient method.

The intuition behind data efficiency differences comes down to the parameter budget. Methods with more trainable parameters need more data to prevent overfitting: if you have 16 million parameters and only 100 examples, the network can trivially memorize the training set without learning generalizable patterns. Methods with fewer trainable parameters are inherently more constrained, and this constraint acts as a form of regularization that can help in low-data settings.