PEFT Beyond LoRA (2024) › Limitations

Despite their advances, these methods also had limitations that constrained their applications. AdaLoRA's adaptive rank allocation required careful tuning of importance metrics and pruning thresholds. The method's performance depended on accurately identifying which layers and rank components were most important, and incorrect importance estimates could lead to suboptimal rank allocation. The additional complexity in training dynamics also made AdaLoRA more sensitive to hyperparameter choices compared to standard LoRA.

DoRA's magnitude-direction decomposition added computational overhead compared to standard LoRA. While the additional parameters were relatively few, the normalization operations required for the direction component increased computation during both training and inference. This overhead, while modest, could be meaningful in scenarios where maximum efficiency was critical. The method's benefits were also most apparent for tasks requiring significant adaptations, with smaller gains for simpler fine-tuning scenarios where standard LoRA already performed well.

VeRA's extreme parameter efficiency came with trade-offs in adaptation flexibility. The shared random matrices and layer-specific scaling provided less expressive power than layer-specific low-rank matrices. While this limitation was acceptable for many tasks, some fine-tuning scenarios required more sophisticated layer-specific adaptations that VeRA's shared structure could not capture effectively. The method worked best when adaptation patterns were relatively simple and could be captured through scaling of shared projections.

The additional complexity of these methods compared to standard LoRA also created barriers to adoption. Implementing adaptive rank allocation, magnitude-direction decomposition, or shared random matrices required more sophisticated training infrastructure and deeper understanding of the underlying mechanisms. Organizations with simpler requirements might find standard LoRA's simplicity and maturity more attractive than the incremental improvements offered by advanced methods.

These methods also did not solve some of the deeper limitations of parameter-efficient fine-tuning. Tasks requiring updates with very high intrinsic rank might still struggle with any low-rank approximation, regardless of how adaptively it was allocated. Scenarios where fine-tuning needed to significantly restructure model knowledge or capabilities might still require full-parameter fine-tuning or other approaches. The advanced methods represented improvements rather than fundamental breakthroughs in addressing parameter-efficient fine-tuning's core challenges.