PEFT Beyond LoRA (2024) › The Problem

Despite LoRA's widespread adoption and success, several fundamental limitations became apparent as researchers applied it to increasingly diverse and challenging fine-tuning scenarios. LoRA's fixed-rank assumption meant that every layer received the same adaptation capacity, regardless of its importance or the complexity of updates it required. For a model with hundreds of layers, this uniform distribution of rank often wasted capacity on less critical layers while constraining important layers that might benefit from higher-rank adaptations. Some layers might only need minimal updates, while others required more sophisticated adaptations that a fixed low rank could not capture effectively.

The low-rank approximation itself placed constraints on the types of weight updates that LoRA could learn. By approximating weight updates as $\Delta W \approx BA$, where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times d}$ with small rank $r$, LoRA assumed that meaningful updates could be captured through low-rank matrix products. While this assumption held for many tasks, some fine-tuning scenarios required updates with higher intrinsic rank or more complex structure that low-rank approximations struggled to represent. Tasks involving significant domain shifts or requiring specialized behaviors might need weight updates that could not be efficiently decomposed into low-rank products.

Rank selection in LoRA created a trade-off that was difficult to optimize. Using too low a rank limited the model's ability to learn necessary adaptations, potentially constraining performance on challenging tasks. Using too high a rank increased memory and computational costs, reducing the efficiency benefits that made LoRA attractive. Finding the optimal rank typically required extensive hyperparameter search across multiple training runs, adding complexity and computational overhead. Different layers might benefit from different ranks, but LoRA's uniform approach could not accommodate this variation.

The uniform application of LoRA across all target layers also ignored the heterogeneous importance of different layers in transformer architectures. Attention layers and feed-forward networks might require different adaptation strategies, and different positions in the model might contribute differently to task-specific behaviors. Early layers might need more adaptation for input processing, while later layers might require fine-tuning for output generation. LoRA treated all layers equally, missing opportunities to allocate adaptation capacity more strategically based on each layer's role and importance.

Additionally, LoRA's weight updates were constrained to additive modifications of the form $W' = W + BA$. While this additive structure simplified implementation and enabled efficient merging of adapters, it might not always be the most effective way to update weights. Some adaptations might benefit from more sophisticated update patterns that could capture multiplicative interactions or other non-additive modifications. The field needed methods that could learn richer update patterns while maintaining parameter efficiency.