PEFT Comparison › Limitations and When to Use Full Fine-tuning

Despite their efficiency, PEFT methods have inherent limitations that should inform your choice about whether to use them at all. Understanding these limitations is as important as understanding their benefits, because deploying a PEFT method in a situation where it will fail creates more problems than it solves.

The most basic limitation is constrained adaptation capacity. By design, PEFT methods limit what the model can learn. For tasks requiring substantial behavioral change, such as learning a new language, acquiring domain expertise in a highly technical field, or fundamentally shifting the model's output format and style, full fine-tuning may be necessary. LoRA's low-rank constraint assumes task-specific modifications live in a low-dimensional subspace, which may not hold for all tasks. When you're trying to teach a model a new programming language it was never exposed to during pre-training, or when you're attempting to align a base model for safety properties, the limited parameter budget of PEFT methods may simply be insufficient to encode the necessary knowledge and behavioral patterns.

A second important limitation is sensitivity to hyperparameters. PEFT methods introduce new hyperparameters (rank, prefix length, bottleneck dimension) that require tuning. Poor choices can materially impact performance, sometimes more sharply than learning rate selection in full fine-tuning. The rank parameter in LoRA is particularly sensitive: a rank that is too low will leave performance on the table, while a rank that is too high may cause overfitting on small datasets. Finding the right rank often requires running multiple experiments, which partially offsets the efficiency benefits of PEFT if your compute budget for experimentation is limited.

Degradation at scale is a third limitation that becomes particularly needed as systems grow more complex. While PEFT methods work well for single-task adaptation, repeatedly applying multiple LoRA adapters or stacking adapters can lead to interference effects. When two adapters have been trained on conflicting objectives, combining them produces unpredictable behavior that may be worse than either adapter alone. Research on LoRA merging and multi-task adapter composition remains an active area with no fully satisfying solution.

PEFT methods also do not replace pre-training. They adapt existing capabilities; they don't create new ones. A model that doesn't encode knowledge about a topic from pre-training generally won't learn it from PEFT alone, regardless of how much task-specific data you give. This is why PEFT works better on tasks that are already related to the model's pre-training distribution than on tasks that require fundamentally new knowledge. If your task falls outside the model's pre-training domain, you may need either a different base model (one pre-trained on needed data) or full fine-tuning with domain-specific data at a scale that can modify the model's knowledge encoding.

When should you opt for full fine-tuning despite the efficiency benefits of PEFT?

• Maximum performance is non-negotiable: In high-stakes applications where even a 1-2% gap matters, full fine-tuning remains the gold standard for squeezing out the best possible performance.
• Sufficient compute is available and cost is secondary: If you have the resources and the time, full fine-tuning removes uncertainty about whether a PEFT method's constraints are limiting your results.
• The task requires deep behavioral change: Domain adaptation, safety fine-tuning, developing a new capability, or adapting a model to a radically new domain may exceed PEFT's adaptation budget.
• You're creating a foundation for further PEFT: Full fine-tuning a domain-specific base model (for example, training a biomedical language model from a general LLM) and then using PEFT for specific downstream tasks is a powerful two-stage strategy that combines the strengths of both approaches.