PEFT Motivation › Limitations of PEFT

Despite its advantages, PEFT has practical limitations that you should understand before committing to it for a production system.

The most significant limitation is the expressivity ceiling. Because PEFT methods restrict the optimization to a small subspace of the full parameter space, there exist tasks for which no configuration of PEFT parameters will achieve performance parity with full fine-tuning. Tasks requiring deep domain adaptation, such as fine-tuning a general-purpose model to produce coherent medical literature or highly structured legal language, may fall into this category. The frozen base model's internal representations may not be organized in a way that small adapters can reorient efficiently for these demanding use cases. When performance is critical and you have the resources, full fine-tuning remains the more reliable choice.

A second limitation is the hyperparameter sensitivity of PEFT methods. While full fine-tuning requires careful tuning of learning rate and batch size, PEFT adds additional hyperparameters like rank, scaling factor (alpha), and target modules that must all be set appropriately. A poorly configured LoRA run with a rank that is too low or an alpha that is too large can perform substantially worse than a well-configured full fine-tuning run. In practice, this means PEFT often requires more experimentation than a naive analysis would suggest, and the community-standard defaults (such as rank 8 and alpha 16) may not be optimal for your specific task.

Third, PEFT methods can sometimes interfere with the base model's capabilities in unexpected ways. When adapters are added to selected layers or weight matrices, they can shift the model's behavior in ways that are difficult to predict from the training loss alone. Catastrophic interference, where the adapter's optimization degrades performance on the base model's original capabilities, is less common with PEFT than with full fine-tuning, but it is not entirely absent. If you need to maintain strong general language understanding alongside task-specific performance, careful evaluation of base model capabilities is warranted even when using PEFT.

Finally, PEFT does not eliminate the computational cost of the forward pass through the frozen base model. The inference cost for a PEFT model with a frozen 70B base model and small adapters is nearly identical to the inference cost for a fully fine-tuned 70B model, since both require running the same number of transformer computations. PEFT reduces training cost and storage cost dramatically, but it does not reduce inference cost in proportion. If your bottleneck is serving latency or throughput rather than training cost or storage, PEFT's advantages may be less relevant to your decision.